2

my Raspberry Pi 3 is currently suffering from a very high IOWait. Running iotop gives me the following: screenshot of iotop

It's listing some perl process but I have no idea what it is or why it's using 99% IO. Any help would be appreciated.

Update with info from comments:
I am using Raspberry Pi OS Buster. This version of Raspberry Pi OS is unmodified. Things I have installed: nginx, grafana, collectd, influxdb, minio. The command I ran was just iotop. I have no clue what this perl process could be.

5
  • 3
    Please don't post pictures of text. Instead paste the text direct into the question. What operating system do you use? What version? Have you done some modifications or is it running unmodified from the image?
    – Ingo
    Jul 9, 2020 at 10:38
  • Sorry, I will keep that in mind for the future. I am using Raspberry Pi OS Buster. This version of Raspberry Pi OS is unmodified. Things I have installed: nginx, grafana, collectd, influxdb, minio
    – Cx24
    Jul 9, 2020 at 10:53
  • Even more importantly, does the image shows a complete command or a truncated command line? It's even more peculiar that your system runs a heavy Perl job as root. Jul 9, 2020 at 10:57
  • The command I ran was just iotop. I have no clue what this perl process could be.
    – Cx24
    Jul 9, 2020 at 11:01
  • Please don't answer your question within the question. It is not the way this site works. Instead create an answer (just cut and paste the UPDATE to it). There is nothing wrong by answering your question. Please accept then the answer after two days. Only accepting an answer will finish the question and it will not pop up again and again, annoying us for years.
    – Ingo
    Jul 10, 2020 at 9:11

3 Answers 3

3

Since you know the PID of the offending process (4076), you should find out what started it. Run ps -aef --forest (as root, because it's a root process) and search for the PID in the process tree.

The keyword automake suggests it could be a software package being installed from source code.

Unless you observe this behavior right after a reboot, check out your .bash_history file or whatever. It could be that you have started a software build and then forgot about it.

5
  • 99% IO is now being used by [kworker/u8:0+flush-179:0] which seemingly doesn't make any sense to me. I tried the command you suggested but apparently nothing started this. Any ideas? Thanks!
    – Cx24
    Jul 9, 2020 at 13:50
  • automake definitely = en.wikipedia.org/wiki/GNU_Autotools So the ultimate cause here was probably something where you ran ./configure then make (or make install).
    – goldilocks
    Jul 9, 2020 at 13:57
  • @Cx24 kworker is a kernel thread which is processing system requests. It won't hog your system unless there's an application sending those requests. If you suspect a kernel problem you could try to catch a trace using echo l > /proc/sysrq-trigger but I don't even know if kernel trace is enabled in the Pi stock kernel. From your screenshot I deduce the problem is with perl, and kernel load is just a consequence. Jul 9, 2020 at 14:11
  • The problem probably is not that or any particular process at all.
    – goldilocks
    Jul 9, 2020 at 14:16
  • @DmitryGrigoryev the perl process is oddly completely gone now. However, high IO is still happening. The only two things I can see is jbd2/mmcblk0p2- and [kworker/u8:0+flush-179:0] but also collectd occasionally. I should probably mention that I have an external HDD connected to the Pi via USB mounted with fstab. No clue if my external HDD could somehow be the cause for all this.
    – Cx24
    Jul 9, 2020 at 14:19
2

The questioner updated his question with the solution. It seems he doesn't like to create a correct answer as suggested in a comment. Here is it:

After spending a lot of time I finally found the root cause. Collectd auto enabled this on install: https://collectd.org/wiki/index.php/Plugin:RRDtool which caused the whole system to use 100% IO for that. Took me way longer to figure out than I'd like to admit. Anyway, my Pi finally seems to be fixed and be back to normal.

0

That automake process is not necessarily the cause of the problem, it may instead be mostly a symptom.

99% IO is now being used by [kworker/u8:0+flush-179:0] which seemingly doesn't make any sense

First, WRT iotop, the IO column shows the percentage of the process's running time that is currently spent waiting on IO.1 So 99% for one (or some, or all) process(es) makes sense, it is just concerning.

[kworker/____] indicates a kernel thread. If anything is busy with IO, it is probably the kernel (it could by necessity mean the kernel in the context of iotop, that would take some more investigation).

As already mentioned WRT this question, automake is a component of Autotools, a suite of things used in configuring and executing a software build from source code (usually written in C and/or C++). If you've done something like this:

cd [some_source_direcory]
./configure
make [install]

That's what started that. If automake and kworker are alternating in iotop I would guess they are related to each other (ie., the kworker is doing stuff for the userland process, automake).

In any case, it may be a bit of cause for concern if you have processes constantly waiting on IO this way and your total disk access is ~32 K/s. It should be a lot faster than that.

This may be because the SD card is wearing out/has developed a defect. You should have a look at dmesg output and int /var/log/syslog to see if any IO errors are being issued.


  1. This implies a sampling period that is presumably quite short. If you want to dive deeper I'd start with that U&L link.
1
  • Thanks for your response. My best guess would be that some other process started to compile something. I never compiled anything on my own though. The perl process is gone by now, but jbd2/mmcblk0p2- and [kworker/u8:0+flush-179:0] and collectd still use way too much IO, which slows down the whole OS. My best guess would be that my external HDD is causing this or collectd doing something I'm not aware of.
    – Cx24
    Jul 9, 2020 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.