Using raspi-config you can decide whether the pi should boot into a desktop environment or not. I only use my pi over ssh so if the desktop is there or not does not matter to me at all.
My question would be: Do I take a noticeable performance hit, with the desktop enabled, when I'm not using any extra graphical applications?
If no one is actually using the desktop, you're not going to notice a direct performance hit, since it is not contending for the CPU -- it's idle.
However, it will occupy a chunk of memory. This may or may not be significant depending on what else is going on. To have a look at how much RAM you are actually using, check free; the important number is the second one on the second line (-/+ buffers/cache -> free). If that's at least half of the total on the first line, don't worry.
If you are using most of your available RAM, performance will suffer because the system uses whatever is left over for page/file caching; this is why output from free can be a bit confusing (if you don't take caching into account, it may appear that you are always using almost all of what's available). The page cache is a good thing, which is why leaving 100 MB or so for it on the pi is beneficial.
Personally, I don't care about graphical boots or logins; if I want to use the GUI, I just log in on a console and use startx. If you generally aren't using the desktop, you might as well disable the display manager service. On raspbian wheezy, presuming you haven't changed the default runlevel (2):
sudo update-rc.d lightdm disable S 2
Or you could just use S 2 3 4 5 to be sure.
If you are interested in which processes occupy the most memory, install and run top, hit shift-F, go down arrow one to select %MEM, hit s then ESC and the stuff at the top of the list is it; RES is the important number. X, the foundation of the GUI stack, may use 75-100 MB all by itself.
