If no one is actually using the desktop, you're not going to notice a direct performance hit, since it is not contending for the CPU -- it's idle.
However, it will occupy a chunk of memory. This may or may not be significant depending on what else is going on. To have a look at how much RAM you are actually using, check
free; the important number is the second one on the second line (
-/+ buffers/cache ->
free). If that's at least half of the
total on the first line, don't worry.
If you are using most of your available RAM, performance will suffer because the system uses whatever is left over for page/file caching; this is why output from
free can be a bit confusing (if you don't take caching into account, it may appear that you are always using almost all of what's available). The page cache is a good thing, which is why leaving 100 MB or so for it on the pi is beneficial.
Personally, I don't care about graphical boots or logins; if I want to use the GUI, I just log in on a console and use
startx. If you generally aren't using the desktop, you might as well disable the display manager service. On raspbian wheezy, presuming you haven't changed the default runlevel (2):
sudo update-rc.d lightdm disable S 2
Or you could just use
S 2 3 4 5 to be sure.
If you are interested in which processes occupy the most memory, install and run
top, hit shift-F, go down arrow one to select
%MEM, hit s then ESC and the stuff at the top of the list is it;
RES is the important number.
X, the foundation of the GUI stack, may use 75-100 MB all by itself.