I've read that 17 LEDs can be controlled individually with the 17 GPIO pins.
Does that mean that only 5
(17 / 3 = 5.6) RGB LEDs can be controlled with one Pi?
Would it be possible to individually control 20 or more RGB LEDs somehow?
Actually, with 17 GPIOs you can drive 72 LEDs using time multiplex (8 groups of 9 LEDs are driven one after another for such a short time that our eyes believes they are on at the same time, just less bright). Do you can have 24 RGB LEDs this way.
And if that's not enough, you can cascade I2C LED drivers resulting in hundreds of LEDs if you want.
Yes and no. If you are talking about discrete LEDs then yes each one will need 3 pins and a common power or ground connection. However, they also make individually addressable RGB LED strips, rings etc. like these Neopixels from adafruit (based on the WS2812), which only require one pin besides power and ground.
Using LED modules that include the WS2812 controller, you can drive as many as you want from a single pin. You can get them in circles, grids and strips, or individual modules.
See this library https://github.com/jgarff/rpi_ws281x
using 3 6:64 latches, driven by 6 gpio for the address lines, 2 gpio for the chip select lines and 1 gpio for the desired new state you can control 64 RGB LEDs I.E. using only 9 gpio lines.
By increasing the number of gpio being used for the chip select lines you can easily control even more LEDs. I.E. double the number of LEDs for each additional gpio chip select line. However by routing those first 6 gpio (for chip select) into a 6:64 latch, (and adding another GPIO for this new latch enable) you can control 64*64 (4096) RGB LEDs using only 10 gpio