The RP does not have a strong Floating points support
It has single precision hardware acceleration, if you use a distro compiled for it, which there are more than a few now. Looking at this, I don't see any reason why the pi should not handle openBTS ("...basically, any computer should do the work. The only thing which is really required is a USB port..."); the guy is using a single core 2.4 Ghz machine with 1 Gb ram, which is considerably faster than the pi, but nothing special.
the amount of memory available to the application after the OS is installed is pretty limited
This is a serious misconception about how operating systems work. Sans GUI, linux needs very very little memory. Here's my raspbian pi right now:
total used free shared buffers cached
Mem: ->448708<- 429392 19316 0 20012 383824
-/+ buffers/cache: ->25556<- ->423152<-
Swap: 102396 0 102396
Those are kB. I've indicated the important numbers. Total 448708 - 423152 free = 25556 used. 25 MB out of 448; this is with the (currently untouched) swap configured as a compressed ram disk, if you change that there should be 512. It's mostly idle right now but it does have a few things running and loaded (eg, ssh and a network sound server).
Obviously, you will hit some kind of limit WRT to the number of calls it can handle, but of course that is true for all machines.
Compiling openBTS may take a while, but all its dependencies should be available as binaries.