Video output filter

Question

I would like to be imbed a filter in the Pi’s HDMI output such that pixels are slightly rearranged. Specifically, I’m after a retro pincushion effect ([ ] -＞ ( )). I need it to be active even in the console/boot sequence, thus X11 mods are out.

I realise this is not an easy task, and that I may end up having to modify the DRM or V4L2 driver in order to achieve this. That’s something I would be willing to do if it came to that.

Does anyone have an idea as to how I might be able to go about this? It’s for a Pi Zero and it doesn’t necessarily need to work with a GUI environment.

Answer 1

You should start with the "Hello Triangle" sample code that comes with Raspbian and works on all models of Raspberry Pi to get started with OpenGL. Just get rid of the spinning and 5 of the 6 cube faces.

To capture the linux framebuffer you can take fb2png apart. Just focus on the parts that acquire the framebuffer /dev/fb0 and convert it into a 32-bit color pixel array and skip the PNG part.

In OpenGL use it to create a texture. If you are confident with double buffering you can use a non-blocking approach. Each frame you will have to update the content of the texture using glTexSubImage2D (I don't recommend releasing and creating a new texture). If you are trying to increase performance, you can try to find the differences in the previously captured and the newly captured linux framebuffer and update only a rectangle instead of the whole texture.

To test your program you should launch it detached by calling it like ./myprog &. This way the program creates the OpenGL context and overlays the screen but the console still is in control of the keyboard. To stop your program type fg on the console to pass keyboard control back to the OpenGL program and then hit [ctrl]+[c] to terminate it.

If you have it working the way you like it you can launch it on boot by creating a systemd service. I'm not sure about which dependencies it might have on boot time so this will be some trail and error to load it as soon as possible during startup.

Answer 2

It always depends on what you put in and expect to get out. So what is your video source? Assuming a USB-Webcam as your V4L2 suggests makes it very CPU intensive. Also doing the deformation on the CPU will limit you in the resolution you can use as the amount of work scales with the pixel count.

The green field for something like that would be to stay on VC4 and do as much hardware accelerated as possible. So the only inputs would be MPEG-4/AVC video streams or the official Raspberry Pi camera. Then you can use the video and pass it over to OpenGL where you can do your cushion deformation in geometry or in a fragement shader.

Leaving that narrow green flield will result in higher CPU usage and lower than realtime frame rates.

Answer 3

I think if it was me, given the constraints, I'd define a new pixel-mangling encoder in vc4 drm that used the writeback engine ("txp") to capture HVS scenes to memory, and make sure that your console and X11 were displaying to that and not HDMI.

Once each write back completes, I'd do whatever algorithm on the captured scene into a new buffer, and then emit pageflips to your HDMI (or whatever) encoder you want to actually see.