I have a Pi Running the OpenElec Version and have tried RaspBMC and also RasPLex (the Plex build for the PI). I've also run XBMC on PC's and laptops. OpenElec version offered the fastest most fluid response for me and will be way easier to set up and get running than Raspbian or Arch Linux.
My pi is connected to my Media via my NAS on Samba shares. I have around 750 Movies (~1.2TB), ~700GB of TV Shows and ~350G Music.
While the Pi isn't as fast or fluid as a PC, it is certainly better than what you are describing with your current set up. For example:
- If I press "movies" from the main menu it will load for about 3-7 seconds
- Press start on a Movie/TV show it will load for about 3-7 seconds before playback
- Playback of 480p & 720p files has been perfect. I have nothing in 1080p but I've heard it works fine.
- Older divx files play fine as well as all my music files. In short, I haven't seen it stutter, buffer or refuse to play anything. Playback is perfect.
The only issue I have is that XBMC is refusing to recognize new movies as I add them and I haven't figured out why (and yes, they are named correctly and according to conventions).
In regards to a remote, try a Media Center Remote like this one from Amazon http://www.amazon.com/gp/product/B001M56DI0/ref=oh_details_o00_s00_i01?ie=UTF8&psc=1
As the First Reviewer points out on the Amazon page, the remote itself is just OK, but the IR receiver can be used with Harmony remotes (I have the Harmony 200). This remote/IR Receiver works out of the box with every version/install/variant of XBMC I've ever tried. Not bad for $8.00.
Now, with all of that said... have you tried Plex? While it's performance is sluggish on the Pi it pretty snappy on the Roku boxes. Please bear in mind that with Plex you need to have the Plex Server running on another computer somewhere and so that may not be for everyone. Also RasPLex (the Pi build for Plex) is still new/beta and so performance may increase as the project matures.
In regards to price, while the Pi board is only $35, you will also need a case, an SD Card, a Power Supply and the cables. You may have some of this laying around and you can make a case from Legos or something if you want. If not, the true cost of the PI is probably going to be closer to $75 which is in the range of several other options. That is still a bargain and if you are something of a geek then a Pi will never go to waste if you decide to move your media center to a different device down the road.
Finally, please bear in mind that this landscape is changing very quickly and there are already a lot of options. I am looking at some of the Android Boxes like the Xios DS or the upcoming Ouya boxes. I'd love to hear feedback from anyone who has tried them and how they compare.