Well, it's not safe if any filesystem present on SD card is mounted (especially if it's the root filesystem that is mounted). This is true by default in most distributions currently available for Raspberry Pi. But it is not strictly needed. You can run your system from RAM (either by copying filesystem image to
tmpfs mounted filesystem or by using
initramfs) or some external harddrive or even from network. If you do so, your SD card can be swapped without problems (just ensure to unmount all filesystems mounted on it).
The only problem is that Raspberry Pi has a small amount of RAM, so if you plan on running it this way, you should have a really small image (compressed filesystems like SquashFS may be handy here), memory split that gives ARM most RAM and probably 512 MB version of Raspberry Pi. But that's definitely possible.