6

I have had good success on the RPi 3 using ffmpeg to decode h.264 streams from various Ethernet video sources. However, on the RPi 4, when I include the -c:v h264_mmal directive, it appears to generate blank frames, as if the decoder never outputs any data. I have re-compiled ffmpeg from the latest source, using the --enable-mmal flag, but it still only outputs what appear to be blank frames. Example output below.

Also, if I use ffplay to play the same video, it plays fine using the software decoder (but uses high CPU, due to software decoding), but when I add the -c:v h264_mmal tag to enable hardware decoding, I get nothing (never loads a video window, etc). (no error message, however)

However, omxplayer does successfully play the video and seems to be using hardware-assisted decoding, as the CPU usage is much lower than ffmpeg on software decoder.

What am I doing wrong? How can I get ffmpeg to work with the hardware decoder?

Example output of ffmpeg (generating snapshots of a video stream)

ffmpeg -c:v h264_mmal -i "rtsp://192.168.1.87/stream0:554?username=admin&password=admin" -vf fps=1/60 img%03d.jpg
ffmpeg version git-2020-02-20-56df829 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --arch=armel --target-os=linux --enable-gpl --enable-omx --enable-omx-rpi --enable-nonfree --enable-mmal
  libavutil      56. 41.100 / 56. 41.100
  libavcodec     58. 71.100 / 58. 71.100
  libavformat    58. 38.101 / 58. 38.101
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 76.100 /  7. 76.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[udp @ 0x36e1fc0] attempted to set receive buffer to size 393216 but it only ended up set as 327680
[udp @ 0x36f2310] attempted to set receive buffer to size 393216 but it only ended up set as 327680
Input #0, rtsp, from 'rtsp://192.168.1.87/stream0:554?username=admin&password=admin':
  Metadata:
    title           : RTSP/RTP stream from anjvision ipcamera
  Duration: N/A, start: 0.083000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, progressive), 1280x720, 12 tbr, 90k tbn, 180k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_mmal) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A 

speed=N/A  frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A  frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A 
...
snip
...
speed=N/A  frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A  [h264_mmal @ 0x381f0c0] Did not get output frame from MMAL.
    Error while decoding stream #0:0: Unknown error occurred
    Finishing stream 0:0 without any data written to it.
    [mjpeg @ 0x3718b30] bitrate tolerance 4000000 too small for bitrate 200000, overriding
    Output #0, image2, to 'img%03d.jpg':
      Metadata:
        title           : RTSP/RTP stream from anjvision ipcamera
        encoder         : Lavf58.38.101
        Stream #0:0: Video: mjpeg, yuvj420p(pc), 1280x720, q=2-31, 200 kb/s, 0.02 fps, 0.02 tbn, 0.02 tbc
        Metadata:
          encoder         : Lavc58.71.100 mjpeg
        Side data:
          cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
    frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x
    video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
    Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
    ^C
2
  • 3
    The GPU has been switched from VC4 to VC6. The Raspberry Pi 4 was a bit rushed such that most of the hardware features from VC6 have not been implemented yet. To my best knowledge I have not seen a Raspberry Pi 4 being able to hardware decode or encode video. This is a software issue. I'd suggest to stick to any other model where the software has been adapted to use the hardware capabilities.
    – kwasmich
    Feb 21 '20 at 15:38
  • 4
    Thanks @kwasmich for this info. It's discouraging considering the length of time the RPI 4 has been out, but at least it's good to know what's up. However, how does this account for omxplayer seeming to use hardware decoding? The CPU usage of omxplayer was around 1%, whereas ffmpeg on software decoding was 25%. That's a huge difference, so omxplayer must be using hardware, right? Feb 21 '20 at 15:40
1

As of July 2021 you can get half way there. There is a V4L2 M2M (memory to memory) interface that is used to access hardware encoders and decoders, and the Pi seems to have pretty good support for that now. So you can do hardware decoding like this:

ffmpeg -c:v h264_v4l2m2m -i file.mkv ...
ffplay -codec:v h264_v4l2m2m file.mkv

The problem is that the decoder can only output a handful of pixel formats, with YUV and NV12 being the only ones that really work. In order to draw the video onto the screen though, it needs to be in RGB. This conversion still happens in software, so uses quite a lot of CPU.

The Pi engineers tell me that currently only OpenGL has hardware support for converting between these pixel formats, so in order to play a video with low CPU use you'd need something that can use OpenGL to render YUV-format images.

If you're just using ffmpeg to convert between formats and you're not playing the video then this is no problem and hardware encoding and decoding works fine, using the *_v4l2m2m codecs (where supported by the Pi hardware).

ffmpeg -codecs | grep v4l2m2m will list the available codecs, not all of which are supported at this time, or by the hardware. (e.g. there's an MJPEG M2M codec, the hardware supports MJPEG, but there's no interface yet linking the M2M interface with the Pi's MJPEG hardware codec so this is currently unavailable.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.