Should playing uncompressed RGB video files consume strictly less resources than playing encoded files?

Question

I don't know much about video, but to play an h264 encoded file into a screen, don't you have to decode it to something like RGB first? I would expect playing a video that is in RGB format to skip the decoding step, therefore being strictly faster than a video that has to be decoded (even if I decode it with hardware acceleration, as I did). That doesn't seem to be the case, but I can't tell whether that is due to a software issue, like the RGB file being played inefficiently. I must be very wrong about something.

I've been playing around all day with GStreamer trying to smoothly play video files on my Raspberry Pi 3. I finally managed it in an almost fresh install of Jessie (after apt-get update, apt-get upgrade and apt-get install gstreamer1.0-tools).

I have an uncompressed (8-bit RGB) .mov version of the video I want to play, and an h264-encoded .mov version. I played both of them:

gst-launch-1.0 filesrc location=h264_test.mov ! qtdemux name=demuxer ! h264parse ! omxh264dec ! ximagesink

gst-launch-1.0 filesrc location=uncompressed_test.mov ! qtdemux ! videoparse format=GST_VIDEO_FORMAT_RGB height=800 width=480 ! glimagesink

Honestly, I expected the uncompressed version would play faster, but it's slow. The h264-encoded version, which omxh264dec decodes with hardware acceleration, plays at the correct speed (which in this case is 25 FPS), but the other one lags and I get GStreamer warnings about buffers being dropped and the computer being slow. I used different sinks for each one (ximagesink and glimagesink) because neither file would play correctly on the other one, which I'm not sure makes a difference.

I think I would've preferred that the uncompressed version was faster, because that means I don't have to bother with fixing banding issues. Really, though, I'm asking this question because I feel like I have a misconception about how this works.

UPDATE: I tried adding a queue, which I understand should buffer video, thinking that SD card reading was the bottleneck:

gst-launch-1.0 filesrc location=uncompressed_test.mov ! qtdemux ! videoparse format=GST_VIDEO_FORMAT_RGB height=800 width=480 ! queue max-size-bytes=125000000 max-size-time=5000000000 max-size-buffers=1000 ! glimagesink

As far as I can tell, the file is playing just as slowly, and I'm still getting GStreamer warnings. Maybe the OpenGL renderer is slow? I'll script some read speed tests later to see how long exactly it takes to read the file.

The particular file I'm using is 116 MB, 4 seconds long and has a bitrate of 232 Mbps.

Answer 1

Since you didn't mention enabling the open source stack, I suspect you're using the bcm video drivers and gstreamer's GL will be using Mesa software rendering instead of hardware. To get a fair comparison you'd need to enable the open source display stack in raspi-config (or use debian or fedora instead)

With glimagesink, gstreamer uploads the decompressed frame into a GL texture, then draws it to the screen. Because it's GL, you get vblank synchronization like any other GL app. Your copies of the video data are (texture, framebuffer) if you're full screen pageflipping or (texture, framebuffer, screen) if you're windowed. Probably add another copy if you're windowed and compositing. (edit: sorry! there's one more copy unless you're using Mesa master because hadn't implemented an important fastpath)

With ximagesink (unless they've added usage of the Present extension, which I highly doubt), you won't get vblank synchronization. Your copies will be (shm, screen) whether or not you're full screen. Add another copy if you're compositing.

Normally glimagesink would be more of a win, not just for vblank synchronization feature, because it uploads YUV data to textures and uses a shader to colorspace convert during the draw, so the upload is smaller and you use less memory bandwidth than the X11 case and a whole lot less CPU for colorspace conversion.