Best way to stream video over internet with RPi?

Question

I'm wondering what is the best way that I can stream video from the camera module over the internet. By "best" I mean lowest latency (can't suffer a second delay for example, this is a real-time control application) & highest FPS without sacrificing much quality. I've read that h.264 could be a good video codec for this under certain profiles (i.e. options), but I am a beginner to these topics.

Here is what I am currently doing with the picamera software (currently using Python, but I am open to using other languages as well for this task), which is currently giving me ~20 FPS at ~160 KB/s over UDP:

with picamera.PiCamera() as camera:
                info("Set up camera")
                camera.resolution = eval(config.get('stream_video', 'picam_resolution'))
                camera.framerate = int(config.get('stream_video', 'picam_framerate'))
                camera.capture_sequence(
                    infinite_seq_of(udp_writer, sleep_time, rfd),
                    'jpeg',
                    use_video_port=True,
                    resize=eval(config.get('stream_video', 'jpeg_resolution')),
                    thumbnail=None,
                    quality=int(config.get('stream_video', 'jpeg_quality')))

Clearly this uses jpeg format, which from my research appears to be inferior to h.264.

Could someone explain / show examples of how I could use h.264 to stream video over the internet, and what trade-offs I need to keep in mind for my application? Cheers

Answer 1

Okay, from your question I'm assuming the following:

• Latency is of primary importance
• Quality (including framerate) is the second consideration
• Compatibility (permitting a wide variety of client software) is irrelevant

Given that latency is king, let's first explore the differences between streaming JPEGs and H.264 and what they mean for latency:

When streaming JPEGs, the camera captures a full frame, encodes it, and dumps it over the net. The client now has a full frame to display (and decoding time is negligible) so the latency is down to the capture, encoding, and transmission time of the frame. In practice, capture and encoding are tiny too, so we'll say it's all down to transmission time.

When streaming H.264 (or any MPEG-ish format in general) things are a tad more complex but not much so. The first frame output by the encoder will be an I-frame (a full frame), so this case is exactly like JPEG (except H.264 I-frames are better quality for the same number of bits): the latency is down to transmission time of the frame. The next frame output by the encoder will most likely be a P-frame (a "predicted" frame) which can't be decoded without knowledge of the preceding frame. All that really means in practice though is that it's much smaller; again, latency is down to transmission time.

What if the prior frame got dropped (you are using UDP after all)? Assuming no re-transmissions, you can't (accurately) decode further frames until the next I-frame. This doesn't exactly affect latency, but in the JPEG scenario it would result in one corrupt frame; in the H.264/MPEG scenario it results in multiple corrupt frames (how many depends on the I-frame period - it's typical to set this quite low when streaming, e.g. once every second or two).

So far, both schemes are looking exactly the same for latency (except H.264 suffers more with dropped packets, but provides better quality). When dealing with camera output that is indeed the case: there's no reason H.264 should have any worse latency than JPEG (and given the smaller transmission sizes it should actually be a bit better). So why does streaming H.264 give poor latency with things like this recipe?

There's another frame-type we haven't considered in the H.264/MPEG scenario: B-frames ("bi-directional predicted" frames). These can't be decoded without knowledge of prior and subsequent frames. Obviously, in a streaming application this will incur latency as you have to wait for the next frame before you can decode the current one. However, we don't have to worry about that because the Pi's H.264 encoder never produces B-frames (I have heard it's common for most video cameras to only produce I-frames and P-frames, which makes perfect sense when you consider they're encoding live material, not pre-recorded stuff where they can look at the next frame to produce a B-frame). So, on the camera side of things B-frames aren't relevant...

But what about the client side? H.264/MPEG players don't know that their source is never going to produce a B-frame. They also assume (not unreasonably) that the network is not reliable and that uninterrupted playback is the primary goal, rather than low latency. Their solution is simple, but effective: buffering. This solves uninterrupted playback and providing space for decoding B-frames based on subsequent frames all in one shot. Unfortunately it also introduces as much latency as there are frames in the buffer!

So, your major issue is on the client side: you need to write/find/coerce client software into forgoing buffering to immediately decode and display incoming frames. Sadly, this is where my knowledge runs out. While mplayer can be cajoled into low latency playback by seeking forwards after playback starts, that's a nasty hack and I'm not aware of ways to accomplish this in other software (VLC, HTML5 <video> tags, etc.), or ways to reliably automate it either.

Incidentally, this is why I deliberately noted compatibility in the list of requirements at the start. Your problem is less to do with the camera (or even the format selected), and more to do with persuading the client to playback with low latency. Whatever solution you come up with is probably going to involve some specific software (or at the very least configuration) on the client. Or to put it another way: getting this working with a specific client sounds doable to me. Getting this working with generic web browsers on a wide variety of disparate platforms? That would be in the "all too difficult" category (at this time).

Links for further reading:

• My own attempts at video streaming using the Pi - this is using a JavaScript based MPEG1 decoder (for wide compatibility) on the client side. If you're considering custom web-browser based decoders, this is probably a good starting point!

• x264 dev blogs about low latency - x264 isn't much use on the Pi (it's way too low power for that) but it's a fascinating read nonetheless.

• Using ffmpeg/gpac to wrap H.264 TS into MP4 - it's worth noting that the Pi's H.264 encoder outputs a raw transport stream that needs wrapping in a MP4 container for most players.

Answer 2

Here is a non-secure but effective way to get low latency from your Pi camera to a windows machine on your local network.

It uses raspivid (standard raspberry pi video capture, and nc (aka netcat) which can (from the command line) send/receive data across the network. On the windows system it uses ncat.exe (aka netcat) to receive the video stream and mplayer to show it. The latency secret is to send 20 frames per second and receive at a higher rate. There is initial latency as the previously buffered frames start coming when the connection is made but quickly get absorbed and the receiver then plays what it gets immediately.

1. Install mplayer on your windows system ( http://sourceforge.net/projects/mplayerwin/ )

2. Install netcat (netc.exe) on windows (http://sourceforge.net/projects/nc110/)

3. Set the path environment variable so the system can find the .exe files, or specify the full path in the invocations shown below.

4. nc is already installed on raspberian

5. activate the camera from raspi-config

Okay now:

1. on the Pi (in this example it is at ip address 192.168.2.108)

   while [ 1 -eq 1 ]; do raspivid -t 0 -h 600 -w 600 -o - -fps 20 -b 5000000 | nc -l 1234; done

This bash command runs a forever loop, starting the raspberry pi video capture program,

'-t 0' implies no timeout,

'-h' and '-w' set the height and width in pixels,

-o -' sets to output to standard out,

'-fps 20' sets the frames per second to 20,

'-b' sets the bitrate.

Then the stdout is piped (the '|') into nc (aka netcat) which sends the frames across the network to whoever connects. The -l argument makes this the listener, awaiting a remote system to connect to it, at port 1234.

2. On the windows system when you want to see the stream, type:

netc.exe -o - 192.168.2.108 1234 | mplayer -demuxer h2643es -fps 60 -

This command line runs netc (aka netcat)

'-o -' output to stdout

and connects to the pi at 192.168.2.108 port 1234.

Then the stdout is piped to mplayer which is told the format is h264es, and to play at 60 frames per second (this eats up the slower stream quickly achieving low latency) and the '-' tells it to get the stream from stdin.

You can terminate the windows side whenever you wish. That terminates the netcat session, but the loop on the Pi side simply starts listening again, ready for the next time the windows PC reconnects.

Answer 3

UV4L is definitely the best way to live stream audio/video over internet. There are plenty of options, ranging from video conferencing to simple intercom, etc...Other ways are just too complex, inefficient or complete sauce, rarely working.