It is my understanding that the USB ports on the Pi share the same controller as the Ethernet port.
This means if one was to use the Pi as a NAS, data transfer over the network would be restricted, since both the USB ports and the Ethernet port would be used simultaneously.
My question is to what effect is data transfer slowed down, and is there a workaround to this problem?
You're right in your understanding that the shared USB/ethernet bus is something of a bottleneck.
One option is to use a Pi 3 over WiFi. The WiFi chip doesn't share a bus with the USB sockets and should run at full speed.
Using any Pi other than a Pi 3, I'm led to believe that using a USB 3 Gigabit ethernet adapter in a USB socket can work wonders for your connection speed. This piece on jeffgeerling.com suggests that he managed to significantly improve network throughput in this way.
The controller is a Microchip LAN9512 and the reason it is used instead of, e.g., a bunch of independent controllers is presumably:
Price. Here's an example of somewhere you can buy 100 of them @ $5 each.
Form factor. You may remember the Raspberry Pi being marketed or tech blogged about as "a credit card sized computer" or "a computer which fits in the palm of your hand", not, "yet another mini-ITX system", or "a computer that will fit under a large hat", etc.
Power. Glancing at the first page of the data sheet (accessible through the Microchip link) "implements reduced power operating modes" is mentioned as one of the "Features". You'd have to dig deeper to find out what that really means in comparative terms but, on the surface at least, it makes conceivable sense.
Remember, the pi cost ~$35 retail. Not $135 or $350. So that's the answer to the literal question in your title, "Why do the USB ports and Ethernet port share the same controller?".
Note it's not an unusual approach for multiple USB ports to share the same bus --- it is probably the norm. I notice the laptop I'm on now has 3 USB ports but lsusb reports only two, 2.0 "Full speed" hubs. So at least two of those ports share a hub and they will not both get "Full speed" at the same time. I'm sure if you check your current system it's put together along similar lines.
Combining the ethernet with that is unusual, but it's worth noting that USB 2.0 should provide 480 Mbps whereas the 10/100 ethernet connection would require at most 100 of those.
Wikipedia notes, citing spec as a source, that, "Due to bus access constraints, the effective throughput of the High Speed signaling rate is limited to 280 Mbit/s or 35 MB/s"; whether that means per port or for the bus as a whole is unclear.
to what effect is data transfer slowed down
Obviously you won't get more than 480 Mbps total, and you probably won't get more than 280. Personally, I've never seen or heard reports of a pi doing better than 10 MB/s transfer from local ethernet to a USB drive (i.e., 80 Mbps) consistently for normal purposes. However, that number is after the protocols themselves have been handled, which will add a variable but I think small percentage -- with ethernet it may vary greatly depending on the topology and activity of the connected network.
is there a workaround to this problem
If you mean, can I squeeze more water through a pipe than the pipe was made to contain then no, or can I get a pump to work faster or some combination or the two again, no. If you need a faster pump or a bigger pipe, buy a faster pump and a bigger pipe.
USB 2.0 has a maximum transfer rate of 480Mb/sec (Megabits)
Now, the actual transfer rate in the real world is far slower.
My Pi 2 has a maximum of 220Mb/sec (Megabits) throughput over USB.
The maxium the Pi can push over the 10/100 network port is about 90Mb/sec (Megabits)
So:
480 Mb/sec (USB 2.0 maxium throughput)
- 260 Mb/sec (For real-world speeds)
- 90 Mb/sec (For the network port)
-----------
130 Mb/sec (Left over bandwidth)
So after fully loading down the network port, you have 130Mb/sec (Megabits) of bandwidth left.
To play it safe, let's make that 100Mb/sec (Megabits) left.
Therefore, (assuming you've got a Pi 2):
As long as your USB bandwidth doesn't exceed 100Mb/sec (Megabits), you'll be fine. 90Mb/sec (Megabits) of that will be used forwarding it to the network card.
So, as long as things other than the HDD connected to the USB ports use less than 10Mb/sec (Megabits) you'll be operating at the maximum speed the Pi 2 can handle.)
Why do the USB ports and Ethernet port share the same controller?
Because the SoC on pi 1 to 3 doesn't have an Ethernet controller, only has a single USB controller and doesn't have any other interfaces that can easily be used for Ethernet. While Eben was able to have some influence on the BCM2835 design it was ultimately a Broadcom product and his bosses at Broadcom would have been calling the shots, there was only so much he could get away with.
Also remember the Pi was designed to be a cheap computer that kids could learn to program on. The fact it was picked up and used for all sorts of other things as well was just a fortunate side effect.
The LAN9512 (later replaced with the LAN9514 to add more ports and the LAN7515 to provide faster Ethernet) provided a convenient solution to adding an extra USB port and an Ethernet port.
When they upgraded the arm cores the pi2 and pi3 they decided to leave the rest of the chip alone (other than fixing a memory controller bug to allow 1GB of ram). I understand this was motivated by trying to make the upgrade as cheap and low-risk as possible.
Finally they reached the point where they had little choice but to do a major overhaul of the SoC design. Fortunately they had also built up their customer base to the point where such an overhaul was financially viable. With the raspberry pi 4 the USB hub with ethernet chip is gone, replaced by an Ethernet PHY chip driven by a MAC on the new SoC and a USB3 controller driven by the PCIe on the new SoC.
