Yes, it should be safe to remove the X server. I haven't actually tried this on the Raspberry Pi (as I don't yet have one), but I expect that removing the X server, with
sudo apt-get remove xserver-xorg will work just fine.
On Debian and most other sensible Linux distributions, the X client packages do not have dependencies on an X server, precisely so that it is possible to have X clients installed on a machine without an X server. Theses X clients can then run and display remotely, as per your use case, which is actually very common for Linux servers.
You should be able to ssh into your Raspberry Pi with X forwarding from a machine running an X server with
ssh -X <your_Pi's_IP_adress>. X clients started over the SSH on the Pi should get forwarded to your X server and appear locally.