Raspberry Pi Stack Exchange is a question and answer site for users and developers of hardware and software for Raspberry Pi. It only takes a minute to sign up.
Sign up to join this community
my Raspberry Pi 3 is currently suffering from a very high IOWait. Running iotop gives me the following:
It's listing some perl process but I have no idea what it is or why it's using 99% IO. Any help would be appreciated.
Update with info from comments:
I am using Raspberry Pi OS Buster. This version of Raspberry Pi OS is unmodified. Things I have installed: nginx, grafana, collectd, influxdb, minio. The command I ran was just iotop. I have no clue what this perl process could be.
Since you know the PID of the offending process (4076), you should find out what started it. Run ps -aef --forest (as root, because it's a root process) and search for the PID in the process tree.
The keyword automake suggests it could be a software package being installed from source code.
Unless you observe this behavior right after a reboot, check out your .bash_history file or whatever. It could be that you have started a software build and then forgot about it.
automake definitely = en.wikipedia.org/wiki/GNU_Autotools So the ultimate cause here was probably something where you ran ./configure then make (or make install).
kworker is a kernel thread which is processing system requests. It won't hog your system unless there's an application sending those requests. If you suspect a kernel problem you could try to catch a trace using echo l > /proc/sysrq-trigger but I don't even know if kernel trace is enabled in the Pi stock kernel. From your screenshot I deduce the problem is with perl, and kernel load is just a consequence.
Jul 9, 2020 at 14:11
The questioner updated his question with the solution. It seems he doesn't like to create a correct answer as suggested in a comment. Here is it:
After spending a lot of time I finally found the root cause. Collectd auto enabled this on install: https://collectd.org/wiki/index.php/Plugin:RRDtool which caused the whole system to use 100% IO for that. Took me way longer to figure out than I'd like to admit. Anyway, my Pi finally seems to be fixed and be back to normal.
That automake process is not necessarily the cause of the problem, it may instead be mostly a symptom.
99% IO is now being used by [kworker/u8:0+flush-179:0] which seemingly doesn't make any sense
First, WRT iotop, the IO column shows the percentage of the process's running time that is currently spent waiting on IO.1 So 99% for one (or some, or all) process(es) makes sense, it is just concerning.
[kworker/____] indicates a kernel thread. If anything is busy with IO, it is probably the kernel (it could by necessity mean the kernel in the context of iotop, that would take some more investigation).
As already mentioned WRT this question, automake is a component of Autotools, a suite of things used in configuring and executing a software build from source code (usually written in C and/or C++). If you've done something like this:
cd [some_source_direcory]
./configure
make [install]
That's what started that. If automake and kworker are alternating in iotop I would guess they are related to each other (ie., the kworker is doing stuff for the userland process, automake).
In any case, it may be a bit of cause for concern if you have processes constantly waiting on IO this way and your total disk access is ~32 K/s. It should be a lot faster than that.
This may be because the SD card is wearing out/has developed a defect. You should have a look at dmesg output and int /var/log/syslog to see if any IO errors are being issued.
