SD card speed (IOPS) is awful

Question

For a long time, I'm noticing that my Raspberry Pi 4 is slow in some scenarios, especially when some disk-consuming operations are running.

For example, docker pull something might take around 5-10 minutes for not so big image. At first, it works pretty fast but after the first 10-20 seconds download/extract speed drops dramatically and then goes very slowly. The same behavior for other processes which intensively involve random I/O. Also, I have a monitoring set, from which I can see that during such operations, SD card read/write requests take up to 8-10 seconds(!!!).

The first spike on the below charts happened while performing docker pull/docker build actions (around 18:15). And the second spike is related to the Agnostics (Raspberry Pi Diagnostics) test (around 20:00).

Raspberry Pi Diagnostics test:

Raspberry Pi Diagnostics - version 0.9
Tue Feb  8 19:56:05 2022

Test : SD Card Speed Test
Run 1
prepare-file;0;0;36942;72
seq-write;0;0;35158;68
rand-4k-write;0;0;938;234
rand-4k-read;470;117;0;0
Sequential write speed 35158 KB/sec (target 10000) - PASS
Random write speed 234 IOPS (target 500) - FAIL
Random read speed 117 IOPS (target 1500) - FAIL
Run 2
prepare-file;0;0;2011;3
seq-write;0;0;1777;3
rand-4k-write;0;0;3;0
rand-4k-read;50;12;0;0
Sequential write speed 1777 KB/sec (target 10000) - FAIL
Note that sequential write speed declines over time as a card is used - your card may require reformatting
Random write speed 0 IOPS (target 500) - FAIL
Random read speed 12 IOPS (target 1500) - FAIL
Run 3
prepare-file;0;0;1857;3
seq-write;0;0;11574;22
rand-4k-write;0;0;10;2
rand-4k-read;1;0;0;0
Sequential write speed 11574 KB/sec (target 10000) - PASS
Random write speed 2 IOPS (target 500) - FAIL
Random read speed 0 IOPS (target 1500) - FAIL
Test FAIL

Yes, those tests and observations were done on the running system, not a fresh installed/formatted, but nothing heavy in disk-consuming terms is running in my system. As you might see, the average number of SD card read/write requests is around 6-9 per second, which is a miserable amount and should not affect the system that much.

Earlier I was using Samsung EVO Plus 64 GB microSDXC Class 10 UHS-I U3 (yes, U3. Not U1 cards which are being produced now instead of U3 by Samsung). Then I started blaming it (faulty, fake card, etc.) and replaced it with SanDisk Extreme Pro 64GB microSDXC Class 10 UHS-I U3 A2. But now I see pretty the same things. So, it can't be both cards are faulty and I believe both are good and it should be something else.

Any ideas on how to troubleshoot, what to try etc.? That can't be true that the SD card performance is so bad on both different cards. I even started thinking about moving my system to an external HDD/SSD...