A relay does exactly what you're saying: it literally closes a connection to complete a circuit. If you want something more low-profile (not necessarily simpler), you could use a transistor, but you have to be careful. The GPIO pins on the Pi run at 3.3V instead if the usual 5V, so if you configure your transistors incorrectly, you could do some damage.
That said, a transistor is probably the best way to do what you're saying, but you have to consider your application. What voltage do you need to control? Is current always flowing in one direction, or does it have to flow both directions? If the current only flows in one direction, you could use a Bipolar junction transistor, or BJT. If the voltage you're controlling is above 3.3V, you're probably going to use an NPN transistor, as shown on this page. Ignore all the arrows on that page, they point in the direction of electron flow instead of current flow, which is just confusing. If you want a lot of current use a Darlington Array, and be careful to not exceed the limits of your transistor(s).
If you need to switch a voltage where the current will flow both directions, you could use a pair of MOSFETs, but relays might be a more common and simple approach. See this answer for more information.
You've probably already found this, but to control the GPIO pins look up the wiringpi library. Good luck!