I do not use a digital monitor and no TV either with my Raspberry Pi. I would like to remove unnecessary packages to save disk space and memory consumption. I would like to preserve the possibility of forwarding X to an SSH client.
Is it safe to remove the xserver? If so, how do I remove it cleanly and thoroughly?
Yes, it should be safe to remove the X server. I haven't actually tried this on the Raspberry Pi (as I don't yet have one), but I expect that removing the X server, with sudo apt-get remove xserver-xorg will work just fine.
On Debian and most other sensible Linux distributions, the X client packages do not have dependencies on an X server, precisely so that it is possible to have X clients installed on a machine without an X server. Theses X clients can then run and display remotely, as per your use case, which is actually very common for Linux servers.
You should be able to ssh into your Raspberry Pi with X forwarding from a machine running an X server with ssh -X <your_Pi's_IP_adress>. X clients started over the SSH on the Pi should get forwarded to your X server and appear locally.
sudo apt-get remove xserver-xorgand everything seems fine.