I was thinking of trying to run some other OS on my raspberry pi that would be better for a really small screen, as Raspbian and the programs that run on it aren't made for screens the size of mine(800 by 480 pixels). I want to try to run an OS made for handheld devices on it, but I am pretty sure it won't work. So what requirements does an OS need to run on the RPi's hardware?
The answer to your question is simple from a high level, but gets very complicated, very quickly. For a high level of understanding, you'll want to go for something that has words like, "For Raspberry Pi".
If you want to dig into it a little more, an OS will have to meet a few requirements. They sound trivial, but are not.
The OS must work on an ARM processor. Specifically, the models found in the RPi. Not all are interchangeable. The snapdragon in my phone isn't the same thing as what's in my RPi 2.
The OS needs to have a specific set of drivers to operate the hardware on the RPi. This part is pretty customized and includes proprietary pieces. Unless someone took specific interest in supporting an RPi with their build, there's not going to be a lot of plug and play here.
It needs to be relatively light weight. Windows 10 (x86 desktop version) isn't the heaviest thing in the world, but it would easily squish an RPi like the fruit it's named after. It's mighty, but little. The OS will have to account for that.
If i'm not mistaken OS has to support ARM processor. But not any OS will do. Somebody has to make a build for Pi. For example Android runs primarily on phones with ARM processors, but I haven yet seen stable build.
From Linux distros that I know should run on Pi are Pidora, Archlinux, Ubuntu, Ubuntu Mate, RISC OS. Should be more if you google. Here's a site to pick what ubuntu supports what Pi hardware.
If your concern is GUI that fits on a small screen you can check out RISC OS build for Pi. Check out screenshots of it's ui, I think it may look ok on small screens.
There also are things like Kodi (ex XBMC) and RetpoPI (emulation env). I'm not sure if they can be called OSs' on their own or just tools. There is a nice image called TriPi it contains Raspbian + Kodi media centre and RetroPI for games.
Update: I remembered that I read about Windows plans to run on Pi and checked on that, seems like you can get Windows 10 IoT version for Pi from their website now. Not sure about Win10 on 800x480 but since it's a version for the Internet of Things they might have some optimisations.