For future reference of anyone who's looking a detailed answer to this same question, please refer to this post on the Raspberry Pi forums: https://www.raspberrypi.org/forums/viewtopic.php?f=2&t=4668
Summary: The RPi supports H.264 Level 4.1 High Profile. If you select these options on the Video page of handbrake, the Pi should play them.
You should make sure that the audio is encoded with AC3 rather than DTS, or you might run into playback issues: see this question: Difference between H.264 and x264 and how to play it
Cheers.
