RPI 4 - ffmpeg - how to get hardware h.264 decoding to work?

Question

I have had good success on the RPi 3 using ffmpeg to decode h.264 streams from various Ethernet video sources. However, on the RPi 4, when I include the -c:v h264_mmal directive, it appears to generate blank frames, as if the decoder never outputs any data. I have re-compiled ffmpeg from the latest source, using the --enable-mmal flag, but it still only outputs what appear to be blank frames. Example output below.

Also, if I use ffplay to play the same video, it plays fine using the software decoder (but uses high CPU, due to software decoding), but when I add the -c:v h264_mmal tag to enable hardware decoding, I get nothing (never loads a video window, etc). (no error message, however)

However, omxplayer does successfully play the video and seems to be using hardware-assisted decoding, as the CPU usage is much lower than ffmpeg on software decoder.

What am I doing wrong? How can I get ffmpeg to work with the hardware decoder?

Example output of ffmpeg (generating snapshots of a video stream)

ffmpeg -c:v h264_mmal -i "rtsp://192.168.1.87/stream0:554?username=admin&password=admin" -vf fps=1/60 img%03d.jpg
ffmpeg version git-2020-02-20-56df829 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --arch=armel --target-os=linux --enable-gpl --enable-omx --enable-omx-rpi --enable-nonfree --enable-mmal
  libavutil      56. 41.100 / 56. 41.100
  libavcodec     58. 71.100 / 58. 71.100
  libavformat    58. 38.101 / 58. 38.101
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 76.100 /  7. 76.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
[udp @ 0x36e1fc0] attempted to set receive buffer to size 393216 but it only ended up set as 327680
[udp @ 0x36f2310] attempted to set receive buffer to size 393216 but it only ended up set as 327680
Input #0, rtsp, from 'rtsp://192.168.1.87/stream0:554?username=admin&password=admin':
  Metadata:
    title           : RTSP/RTP stream from anjvision ipcamera
  Duration: N/A, start: 0.083000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuvj420p(pc, progressive), 1280x720, 12 tbr, 90k tbn, 180k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_mmal) -> mjpeg (native))
Press [q] to stop, [?] for help
frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A 

speed=N/A  frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A  frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A 
...
snip
...
speed=N/A  frame=    0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A  [h264_mmal @ 0x381f0c0] Did not get output frame from MMAL.
    Error while decoding stream #0:0: Unknown error occurred
    Finishing stream 0:0 without any data written to it.
    [mjpeg @ 0x3718b30] bitrate tolerance 4000000 too small for bitrate 200000, overriding
    Output #0, image2, to 'img%03d.jpg':
      Metadata:
        title           : RTSP/RTP stream from anjvision ipcamera
        encoder         : Lavf58.38.101
        Stream #0:0: Video: mjpeg, yuvj420p(pc), 1280x720, q=2-31, 200 kb/s, 0.02 fps, 0.02 tbn, 0.02 tbc
        Metadata:
          encoder         : Lavc58.71.100 mjpeg
        Side data:
          cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
    frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x
    video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
    Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
    ^C

Answer 1

As of July 2021 you can get half way there. There is a V4L2 M2M (memory to memory) interface that is used to access hardware encoders and decoders, and the Pi seems to have pretty good support for that now. So you can do hardware decoding like this:

ffmpeg -c:v h264_v4l2m2m -i file.mkv ...
ffplay -codec:v h264_v4l2m2m file.mkv

The problem is that the decoder can only output a handful of pixel formats, with YUV and NV12 being the only ones that really work. In order to draw the video onto the screen though, it needs to be in RGB. This conversion still happens in software, so uses quite a lot of CPU.

The Pi engineers tell me that currently only OpenGL has hardware support for converting between these pixel formats, so in order to play a video with low CPU use you'd need something that can use OpenGL to render YUV-format images.

If you're just using ffmpeg to convert between formats and you're not playing the video then this is no problem and hardware encoding and decoding works fine, using the *_v4l2m2m codecs (where supported by the Pi hardware).

ffmpeg -codecs | grep v4l2m2m will list the available codecs, not all of which are supported at this time, or by the hardware. (e.g. there's an MJPEG M2M codec, the hardware supports MJPEG, but there's no interface yet linking the M2M interface with the Pi's MJPEG hardware codec so this is currently unavailable.)