You don't need internet to ssh to your pi... But you do need a connection between the two(a LAN connection). So forget about the cell tethering altogether(as regards ssh at least, they are two separate issues.)
To ssh to your pi you can either get a switch and two standard network cables or one crossover cable or set up a wifi adapter on the pi and an ad hoc network between the pi and the computer.
The crossover cable is probably the cheapest. You can make one yourself if you've got an RJ-45 crimper. Once you have the link, then you set static IPs on both your PC and the pi. (i.e. set your TCP/IP address in windows to 192.168.0.1 and the pi to 192.168.0.2 and use the netmask 255.255.255.0 for both. You don't need a gateway or DNS if there's no internet)
(To set a static IP on the pi see - http://www.modmypi.com/blog/tutorial-how-to-give-your-raspberry-pi-a-static-ip-address. In windows it's under Network and Sharing Center->Change Adapter Settings. Right click the adapter and select properties then Internet Protocol Version 4.)
once you've done that, you should be able to ssh to the pi at whatever 192.168.xxx.xxx address you gave it.
Edit 2015-12-23: Corrected example IP addresses to ones that are on the same (sub)network address!
Edit#2 2015-12-23: Unfortunately the linked method of setting a
static IP address probably won't work on recent Raspbian.