Okay, from your question I'm assuming the following:
- Latency is of primary importance
- Quality (including framerate) is the second consideration
- Compatibility (permitting a wide variety of client software) is irrelevant
Given that latency is king, let's first explore the differences between streaming JPEGs and H.264 and what they mean for latency:
When streaming JPEGs, the camera captures a full frame, encodes it, and dumps it over the net. The client now has a full frame to display (and decoding time is negligible) so the latency is down to the capture, encoding, and transmission time of the frame. In practice, capture and encoding are tiny too, so we'll say it's all down to transmission time.
When streaming H.264 (or any MPEG-ish format in general) things are a tad more complex but not much so. The first frame output by the encoder will be an I-frame (a full frame), so this case is exactly like JPEG (except H.264 I-frames are better quality for the same number of bits): the latency is down to transmission time of the frame. The next frame output by the encoder will most likely be a P-frame (a "predicted" frame) which can't be decoded without knowledge of the preceding frame. All that really means in practice though is that it's much smaller; again, latency is down to transmission time.
What if the prior frame got dropped (you are using UDP after all)? Assuming no re-transmissions, you can't (accurately) decode further frames until the next I-frame. This doesn't exactly affect latency, but in the JPEG scenario it would result in one corrupt frame; in the H.264/MPEG scenario it results in multiple corrupt frames (how many depends on the I-frame period - it's typical to set this quite low when streaming, e.g. once every second or two).
So far, both schemes are looking exactly the same for latency (except H.264 suffers more with dropped packets, but provides better quality). When dealing with camera output that is indeed the case: there's no reason H.264 should have any worse latency than JPEG (and given the smaller transmission sizes it should actually be a bit better). So why does streaming H.264 give poor latency with things like this recipe?
There's another frame-type we haven't considered in the H.264/MPEG scenario: B-frames ("bi-directional predicted" frames). These can't be decoded without knowledge of prior and subsequent frames. Obviously, in a streaming application this will incur latency as you have to wait for the next frame before you can decode the current one. However, we don't have to worry about that because the Pi's H.264 encoder never produces B-frames (I have heard it's common for most video cameras to only produce I-frames and P-frames, which makes perfect sense when you consider they're encoding live material, not pre-recorded stuff where they can look at the next frame to produce a B-frame). So, on the camera side of things B-frames aren't relevant...
But what about the client side? H.264/MPEG players don't know that their source is never going to produce a B-frame. They also assume (not unreasonably) that the network is not reliable and that uninterrupted playback is the primary goal, rather than low latency. Their solution is simple, but effective: buffering. This solves uninterrupted playback and providing space for decoding B-frames based on subsequent frames all in one shot. Unfortunately it also introduces as much latency as there are frames in the buffer!
So, your major issue is on the client side: you need to write/find/coerce client software into forgoing buffering to immediately decode and display incoming frames. Sadly, this is where my knowledge runs out. While mplayer can be cajoled into low latency playback by seeking forwards after playback starts, that's a nasty hack and I'm not aware of ways to accomplish this in other software (VLC, HTML5 <video>
tags, etc.), or ways to reliably automate it either.
Incidentally, this is why I deliberately noted compatibility in the list of requirements at the start. Your problem is less to do with the camera (or even the format selected), and more to do with persuading the client to playback with low latency. Whatever solution you come up with is probably going to involve some specific software (or at the very least configuration) on the client. Or to put it another way: getting this working with a specific client sounds doable to me. Getting this working with generic web browsers on a wide variety of disparate platforms? That would be in the "all too difficult" category (at this time).
Links for further reading: