I'm afraid that the most effective thing you can do is to overclock your Pi (there's an option for that in raspi-config). You may want to put a heatsink on the CPU chip to prevent overheating. This should give you a couple percent performance increase.
Adding swap on a USB HDD will not increase performance at all, since your Pi only supports USB 2.0, which is pretty slow, so using a HDD plugged into that as a 'RAM extension' wouldn't be effective.
Analyze your running processes with htop or top.
Figure out what is installed that you don't need running and disable or remove the packages (not for the feint of heart or linux beginner).
Don't start the gui.
Disable all logging or send logs to dev/null
Make sure the PSU is supplying ample power as others have stated.
You can overclock, but I wouldn't do this without good cooling and heat sinks.
Externally power any addons.
Then you should have as high performance as I could advise.