During the installation of Raspian I chose "Start X-Server after boot", but now I want to boot it once without starting the X-Server, so I've got more memory to run a program.
How do I boot without starting the X-Server?
With the Raspbian image, you can re-run the initial start up script using:
$ sudo raspi-config
and entering your sudo password.
This will bring up the same menu options that you got after first boot.
You do not need to remake all your first boot choices, just use the arrow keys to move to the menu options you want to change.
In your case, selecting:
Start X-server after boot?
no or disable
will sort you out.
Editing to add non gui based options:
As noted in the comments below by @mirk, the exact command for the raspberry is:
sudo update-rc.d lightdm disable
How X.Org works is that you have a desktop manager that boots the X Server for you. That means that in order to prevent X Server from starting, you need to remove your destkop manager from the daemons list.
If you are using GNOME then you're desktop manager is GDM. If you're using KDE you're desktop manager will be KDM. For this, I'm going to assume that you're using GNOME; ergo you are using GDM as your desktop manager.
In Raspian, the desktop manager is called LightDM.
To disable a daemon in Debian (which Raspbian is a fork of) you have to run the following command.
$ sudo update-rc.d <daemon> disable
So in order to disable LightDM we have to run the following command.
$ sudo update-rc.d lightdm disable
And now on your next reboot LightDM shouldn't start and will, in turn, not start X Server.