I realized that my PI is lagging over ssh (Demo: https://www.youtube.com/watch?v=0wkBXIFKHAg) The cause of this turns out to be packet loss (ca 10%) within my home network between the PI and my Desktop, both of which are connected over Ethernet to my home router. The thing is that there are times when there is no packet loss but this seems only to last for a few minutes max, and then the packet loss returns. In order to verify that the PI is at fault, I disconnected it from my router and connected my laptop to the same port. There was no packet loss between my Desktop and my Laptop. Also I just reinstalled a fresh copy of the latest raspbian, connected it again to my router and it still has packet loss. I use the PI 4 and the official power supply. I have no idea what might be the issue. Maybe a defective unit?
I solved that problem by using a netgear switch behind my ISP's router. The "Unitymedia ConnectBox" in my case generated about 15% packet loss. I experienced all the problems you described and it's really strange that the wifi connection runs just fine. Also RPi1 and 2 worked with my router as well (haven't bought the RPi3).
All my network devices are now attached to a NETGEAR GS308-300PES. Even when pinging my Pi from the router there's no packet loss anymore ... hope that helps since there are a lot of threads troubleshooting this problem without a clear answer.
Recently I faced a similar problem.
My issue turned out to be too many splices in the ether cable - I had ~50ft. of cable running through the walls of my house with unknown amount of splices - which caused signal degradation.
When I connected my laptop to this cable, there were minimal to no problems. When I connected my RPi to the same cable, it was not even able to get an IP from the router's DHCP.
One thing you could try: connect you RPi with as short of a cable as possible to your router, and see if the problem persists. This will certainly rule out (or confirm) the possibility of a defective unit.