0

I've been getting extremely slow responses from my Raspberry Pi 4, which I manage headlessly from within my LAN. This includes:

  • Very slow connections to SSH, and commands entered into an SSH terminal taking a very long time to be actioned;
  • Very slow connections to networked drives on my Windows machine;
  • Services on my apache server timing out/taking more than a minute to connect.

I've tried connecting the Pi over ethernet (direct to router), and WiFi, but the problem persists.

Running Raspbian, happy to provide any other information you feel is relevant.

Output for top:

top - 12:33:38 up 13 min,  2 users,  load average: 9.31, 8.25, 5.23
Tasks: 163 total,   1 running, 162 sleeping,   0 stopped,   0 zombie
%Cpu(s):   1.3/0.5     2[|                                                                                                   ]
MiB Mem :   1866.5 total,    336.1 free,    687.4 used,    843.0 buff/cache
MiB Swap:   1024.0 total,   1023.0 free,      1.0 used.   1077.5 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  758 rslsync   20   0  645400 456436   8844 S   5.9  23.9   6:46.12 rslsync
  806 root      20   0 1077036  73936  33192 S   0.0   3.9   0:12.53 dockerd
  738 mysql     20   0  725104  72960  15584 S   0.0   3.8   0:01.78 mysqld
  652 root      20   0  964156  37208  18216 S   0.0   1.9   0:04.10 containerd
 2255 rslsync   20   0   50460  35552   7660 S   0.3   1.9   0:06.36 archivebox
  786 root      20   0  212060  25016  20300 S   0.0   1.3   0:00.24 apache2
  984 root      20   0   26168  19484   3372 S   0.0   1.0   0:00.10 miniserv.pl
  823 root      20   0   48664  16580  14632 S   0.0   0.9   0:00.32 smbd
  629 root      20   0   38712  15888  10756 S   0.0   0.8   0:00.30 unattended-upgr
  645 root      20   0   31984   9508   8020 S   0.0   0.5   0:01.08 nmbd
  116 root      20   0   37648   8808   7916 S   0.0   0.5   0:01.58 systemd-journal
  415 root      20   0   54652   8412   7244 S   0.0   0.4   0:00.41 ModemManager
    1 root      20   0   33880   8108   6392 S   0.0   0.4   0:10.92 systemd
 1738 root      20   0  803524   7764   6028 S   0.0   0.4   0:00.36 containerd-shim
 1203 pi        20   0   14736   7484   6428 S   0.0   0.4   0:05.18 systemd
  931 www-data  20   0  212092   7348   2620 S   0.0   0.4   0:00.00 apache2
  932 www-data  20   0  212092   7348   2620 S   0.0   0.4   0:00.00 apache2
  933 www-data  20   0  212092   7348   2620 S   0.0   0.4   0:00.00 apache2
  934 www-data  20   0  212092   7348   2620 S   0.0   0.4   0:00.00 apache2
  935 www-data  20   0  212092   7348   2620 S   0.0   0.4   0:00.00 apache2
 9669 root      20   0  802756   7128   5792 S   0.0   0.4   0:00.03 containerd-shim
 7347 root      20   0   12240   6260   5464 S   0.0   0.3   0:00.07 sshd
  927 root      20   0   48664   5888   3940 S   0.0   0.3   0:00.01 lpqd
  517 root      20   0   37964   5772   5140 S   0.0   0.3   0:00.12 polkitd
  410 root      20   0   13160   5756   5068 S   0.0   0.3   0:00.47 systemd-logind
  677 root      20   0   10728   5680   5100 S   0.0   0.3   0:00.02 sshd
  370 systemd+  20   0   22416   5592   4940 S   0.0   0.3   0:00.32 systemd-timesyn
  913 root      20   0   45424   5300   3420 S   0.0   0.3   0:00.00 smbd-notifyd
  192 root      20   0   18312   4092   3176 S   0.0   0.2   0:01.19 systemd-udevd
 7424 pi        20   0   12240   4092   3288 S   0.3   0.2   0:00.17 sshd
  914 root      20   0   45416   4068   2188 S   0.0   0.2   0:00.00 cleanupd
  419 root      20   0   10744   3732   3360 S   0.0   0.2   0:00.05 wpa_supplicant
 1247 pi        20   0    8492   3704   2752 S   0.0   0.2   0:00.12 bash
 7427 pi        20   0    8488   3700   2748 S   0.0   0.2   0:00.14 bash
  418 message+  20   0    6668   3616   3016 S   0.0   0.2   0:01.15 dbus-daemon
  576 root      20   0    9536   3400   3116 S   0.0   0.2   0:00.03 bluetoothd
 1539 root      20   0  852936   3348   2188 S   0.0   0.2   0:00.01 docker-proxy
 1559 root      20   0  862156   3220   2044 S   0.0   0.2   0:00.01 docker-proxy
 1322 root      20   0  853896   3132   1956 S   0.0   0.2   0:00.01 docker-proxy
  420 root      20   0   25512   3120   2404 S   0.0   0.2   0:00.42 rsyslogd
 1329 root      20   0  862860   3112   1932 S   0.0   0.2   0:00.01 docker-proxy
  426 avahi     20   0    6024   3104   2640 S   0.0   0.2   0:00.94 avahi-daemon
 9396 pi        20   0   10328   3024   2488 R   0.7   0.2   0:00.67 top
 1204 pi        20   0   35292   2796   1748 S   0.0   0.1   0:00.00 (sd-pam)
  983 root      20   0    5620   2780   2336 S   0.0   0.1   0:00.03 login
  435 root      20   0    7976   2400   2220 S   0.0   0.1   0:00.02 cron
  434 nobody    20   0    4320   2188   2016 S   0.0   0.1   0:00.03 thd
  431 root      39  19    3692    744    636 S   0.0   0.0   0:00.03 alsactl
 1801 root      20   0    1272    316    272 S   0.0   0.0   0:00.05 dumb-init
  448 avahi     20   0    5768    252      0 S   0.0   0.0   0:00.00 avahi-daemon
  561 root      20   0    2140    128      0 S   0.0   0.0   0:00.00 hciattach
  437 root      20   0   27656     80      0 S   0.0   0.0   0:00.11 rngd
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    7 root      20   0       0      0      0 D   0.0   0.0   0:02.62 kworker/u8:0+ext4-rsv-conversion
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
   10 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
   11 root      20   0       0      0      0 S   0.0   0.0   0:00.57 ksoftirqd/0
   12 root      20   0       0      0      0 I   0.0   0.0   0:00.53 rcu_sched
   13 root      rt   0       0      0      0 S   0.0   0.0   0:00.02 migration/0
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1


@SEWTGIYWTKHNTDS Thanks - I thought it might be something like this. I thought it might be something to do with the resilio-sync package, so I tried disabling it, rebooting and re-running top.

With resilio-sync disabled:

top - 11:08:40 up 2 min,  2 users,  load average: 7.14, 2.39, 0.86
Tasks: 180 total,   1 running, 179 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  3.6 sy,  0.0 ni, 21.7 id, 74.7 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1866.5 total,    691.4 free,    399.1 used,    776.1 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.   1253.0 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2814 pi        20   0   10444   3044   2532 R  13.6   0.2   0:00.08 top
    1 root      20   0   33804   8172   6428 S   0.0   0.4   0:05.49 systemd
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    5 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0-pm
    6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
    7 root      20   0       0      0      0 I   0.0   0.0   0:00.16 kworker/u8:0-events_unbound
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq

With resilio-sync enabled:

top - 11:14:22 up 1 min,  2 users,  load average: 5.32, 1.59, 0.55
Tasks: 145 total,   1 running, 144 sleeping,   0 stopped,   0 zombie
%Cpu(s): 27.6 us, 22.7 sy,  0.0 ni, 35.3 id, 13.7 wa,  0.0 hi,  0.6 si,  0.0 st
MiB Mem :   1866.5 total,    600.0 free,    739.0 used,    527.5 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.   1036.7 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  761 rslsync   20   0  781204 582340   8932 S  73.7  30.5   0:13.87 rslsync
  808 root      20   0 1035672  62028  31328 S  12.3   3.2   0:02.16 dockerd
  549 root      20   0       0      0      0 I   6.1   0.0   0:00.26 kworker/u8:3-netns
 1356 pi        20   0    8492   3664   2712 S   6.1   0.2   0:00.11 bash
  110 root       0 -20       0      0      0 I   5.0   0.0   0:00.13 kworker/0:2H-kblockd
  979 root      20   0    5620   2488   2100 S   2.8   0.1   0:00.05 login
    1 root      20   0   33752   8060   6420 S   2.2   0.4   0:04.34 systemd
   83 root       0 -20       0      0      0 I   1.1   0.0   0:00.16 kworker/2:1H-kblockd
  407 root      20   0   13160   5888   5192 S   1.1   0.3   0:00.15 systemd-logind
  430 message+  20   0    6668   3548   3040 S   1.1   0.2   0:00.28 dbus-daemon
  981 pi        20   0   14732   7344   6424 S   1.1   0.4   0:00.28 systemd
   11 root      20   0       0      0      0 S   0.6   0.0   0:00.22 ksoftirqd/0
   22 root      20   0       0      0      0 S   0.6   0.0   0:00.09 ksoftirqd/2
   74 root      20   0       0      0      0 I   0.6   0.0   0:00.13 kworker/0:2-events
   85 root      20   0       0      0      0 S   0.6   0.0   0:00.02 jbd2/mmcblk0p2-
  104 root       0 -20       0      0      0 I   0.6   0.0   0:00.14 kworker/3:2H-kblockd
  121 root      20   0   21368   6772   5876 S   0.6   0.4   0:00.67 systemd-journal
  434 avahi     20   0    5900   2524   2272 S   0.6   0.1   0:00.17 avahi-daemon
 1010 pi        20   0   12240   3428   2632 S   0.6   0.2   0:00.01 sshd
 1329 pi        20   0   10332   3156   2532 R   0.6   0.2   0:00.03 top
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    5 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0-events
    6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
    7 root      20   0       0      0      0 I   0.0   0.0   0:00.02 kworker/u8:0-cfg80211
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_

I also left resilio-sync enabled and ran ifconfig -a twice, a minute apart, as you suggested. Looking at eth0, it only transferred about 700 KiB in 1 minute (both RX and TX): I'm also connected on wlan but that was neglible. All other interfaces were either 0 or pretty much 0.


Screenshot of htop, after disabling Resilio Sync (rslsync), ordered by CPU usage:

enter image description here

According to cat /sys/class/thermal/thermal_zone0/temp, CPU is running at 64.8 degrees celsius.


Output of dstat --top-io:

----most-expensive----
     i/o process
bash        316k  180k
systemd      83k    0
systemd      85k    0
systemd      20k    0
systemd      18k    0

And just plain old dstat:

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  5   5  18  72   0|1516k 3841k|   0     0 |   0     0 | 837  1468
  0   0  25  74   0|   0  7168k|  60B  908B|   0     0 | 254   422
  0   1  25  75   0|   0  5120k|  60B  476B|   0     0 | 213   387
  1   1  25  74   0|   0  5120k|  60B  476B|   0     0 | 551  1059
  0   1  25  75   0|   0  7168k|  60B  476B|   0     0 | 377   660
  5  21  22  53   0|8192B 4392k|  60B  476B|   0     0 |1081  1997
  0   1  25  74   0|8192B 3104k|  60B  740B|   0     0 | 343   592
  0   0  25  75   0|   0  4096k|  60B  358B|   0     0 | 200   375
  0   0  25  75   0|   0  2048k| 146B  476B|   0     0 | 199   356
  0   0  25  75   0|   0  4096k| 132B  476B|   0     0 | 207   371
  0   1  25  74   0|   0  7168k|  60B  476B|   0     0 | 174   378
  0   0  25  75   0|   0  2048k|  60B  476B|   0     0 | 197   369
  0   0  25  75   0|   0  4104k| 628B  476B|   0     0 | 240   411
  0   1  25  74   0|   0  5112k|  60B 1380B|   0     0 | 222   379
  0   0  25  75   0|   0  7168k| 554B  476B|   0     0 | 243   394
  0   0  25  75   0|   0  6144k|  60B  476B|   0     0 | 206   386
  0   0  25  75   0|   0  7168k|  60B  476B|   0     0 | 214   406
  0   1  25  74   0|   0  2048k| 120B  518B|   0     0 | 191   367
  1   2  25  73   0|   0  5128k|  60B 1287B|   0     0 | 210   400
  0   0  25  75   0|   0  5112k|  60B  358B|   0     0 | 209   426

Pastebin of dmesg output is here (https://pastebin.com/f2nxj1my), the only error messages (I assumed the ones in red) were:

[   14.339379] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   14.340387] usbcore: registered new interface driver brcmfmac
[   14.369876] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt failed with error -2
[   14.614200] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   14.626948] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Jan  4 2021 19:56:29 version 7.45.229 (617f1f5 CY) FWID 01-2dbd9d2e
[   16.606941] random: crng init done
[   16.606966] random: 7 urandom warning(s) missed due to ratelimiting
[   16.665394] 8021q: 802.1Q VLAN Support v1.8
[   16.751337] uart-pl011 fe201000.serial: no DMA platform data
[   17.217670] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled

1 Answer 1

1

It sounds like the Pi is very busy, you can check the cpu time and processes using top

top

Hopefully the processes using the cpu will be clear to see

If it is a IO issue you can use iotop to explore the usage

sudo iotop

If this doesn't help then please post the output of

ifconfig -a

netstat -rnv

That should be a good start to provide information to diagnose network configuration problems

18
  • Have posted the output of top in the question body. Looking at it, it seems like archivebox is the culprit... I'm not sure why rslsync (Resilio Sync) is accessing archivebox - do you think those load numbers would point to the issues I'm having? Jul 20, 2021 at 11:36
  • 1
    The Load average is extremely high, I have never seen anything with numbers that high, my "busy" pi 4 has load averages around 0.2. The Top output is missing the time spent waiting, idle and system are any of those numbers high? The iotop output looks like it's not disk io that is the issue. I suggest you look at the amount of data going over the network, ifconfig -a will give a figure, run it twice a minute or so apart and see the difference on each interface. TCPDump would also help see where the packets are going. Jul 20, 2021 at 19:04
  • Thanks - have updated the question with the outputs. Network data doesn't seem to be the issue. Jul 21, 2021 at 11:54
  • I copied everything from the top output, can't see where I might have missed the time spent waiting? Jul 21, 2021 at 13:40
  • The load average is way too high in all of those. In the last one, you can see one core is working 100% (27.6% is of all four), rslsync seems to be the major culprit (73% is of one, top is a bit confusing this way, htop may be clearer). If you don't have pretty serious cooling and this is frequent or constant, performance will be (further) degraded by processor throttling (starts at 80° I think). However, even if you don't have cooling, cooling is not the answer to this problem. Either you are asking too much activity wise or something is going awry, I'd guess the latter.
    – goldilocks
    Jul 21, 2021 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.