There is absolutely no danger in adding a switch in order ton control the fan.
If you're going to add a switch in the current loop, I recommend you to do so on the loop returning to ground.
Indeed if you put a simple switch there is no problem in putting the switch between 5V and the fan. But if one day you want to make something more sophisticated like a PWM controlled fan, you will need a transistor. And this transistor is likely to be a mos-Fet canal-N. Which works better (if not only) if on the fan ---> ground side.
It works better because the voltage needed at the gate to control it will be less
Hope I'm clear.
I've done it on my, which allow me to control the fan with the GPIO, even no need to touch the raspberry !