What are the downsides to overclocking (even on moderate mode) on my Pi other than it has the chance of becoming unstable? Also, if it does get unstable, is there a way to remove it from overclock mode as I reboot the Pi?
Apart from instability? Not much really. You'll use a little more power. Slightly shorten the life of the processor (probably not significant)
Removing overclock is easy if you have a card reader on your PC
If you don't have a card reader, it will depend on the distro. NOOBS has a rescue mode for example
Otherwise you can try holding down shift when it boots - but that assumes you have a keyboard plugged in