top/htop not showing CPU usage of kernel processes

Question

I'm trying to improve FPS from a USB webcam. Supposedly I should be able to use htop to show where the bottleneck is, since top won't show kernel processes such as time spent waiting for disk access. (top only shows about 10% CPU usage when clearly the system isn't coping).

Is there a tool besides htop which would show kernel process usage? Or is there some sort of trick I'm missing?

Thanks

Answer 1

I'm not sure what "kernel processes" you are talking about. There is such a thing as kernel threads aka. kernel daemons (although there are fundamental similarities, in context do not confuse this with process "threads" generally, which exist in userspace).

• https://stackoverflow.com/questions/9481055/what-is-a-kernel-thread
• https://unix.stackexchange.com/questions/31595/are-linux-kernel-threads-really-kernel-processes

Kernel threads usually have names like kworker and kthreadd and have other special characteristics by which they can be identified (read the link) but they are unlikely to have anything to do with what you are doing.

top does show these by default. htop doesn't, but that can be toggled with shift-k (see the htop man page WRT kernel threads).

Kernel threads do not at all represent the totality of what the kernel is doing, most of which is in service to specific userland processes. WRT your example of waiting for disk access, in so far as this time is active processor time, it is reported as part of the time used by the process doing the deed. However, waiting on disk access is not necessarily time spent using the processor. This is time that involves neither kernel nor user space activities so it is not reportable as an activity (because it isn't an activity of the processor), although it can be measured on a clock (the same way a network request-response cycle can be timed on a clock, although it mostly does not involve any kind of activity by the local computer).

Specific I/O bottlenecks are found by inference this way (timing the round trip). I suspect you have it backward when you say, "clearly the system isn't coping"; in fact the system is mostly idle because there is an I/O bottleneck.

You can more generally infer significant IO/hardware related issues if the load average is high while the CPU is relatively idle. Top and htop both report load average and overall cpu usage at the top. A high load average is more than 1.0; if you notice that consistently while the CPU usage is less than 100%, it indicates a lot of waiting on hardware is going on. This could be the hardware itself malfunctioning or the driver which controls it. Diagnosing what hardware exactly is then a matter of stopping and starting whatever processes; eg, if you are doing something with a video camera and writing the data to disk, try doing the same thing but throwing the data away instead (eg, to /dev/null). If the problem persists, it's the camera. If not, it is disk access.

If the load average is less than one and the cpu isn't maxed out, the system simply is not under load, meaning whatever envelope you think you are pushing, you actually aren't.

Answer 2

There is a very important line in top called "Cpu(s)" (3rd line in my top) where you can find where CPU time is being spent:

• us means userspace
• sy means system which can be seen as "time spent on kernel work", usually by it's processes or treads
• ni means nice which is userspace processes with nice value higher than 0, you can add it to us
• id means idle - this is how much CPU time is left
• wa means wait which is CPU time spent on handling I/O (waiting for I/O to complete, etc)
• hi means time spent processing hardware interrupts
• si means time spent on processing software interrupts (it's a time spent by kernel usually to process driver related tasks that are not handled by hardware interrupts)
• st means stolen and is only relevant in virtual machines

So to sum up, if your userspace applications don't use too much CPU (which is indicated by us+ni) then it can be used by kernel processes which is indicated by sy or it can be used for I/O processing which is indicated by wa or it may be used to handle hardware related tasks which is indicated by hi+si.

In htop the same information is showed by colours in CPU usage bar. I find it less readable so I prefer top for reading that. You can hit F1 to see the help page where you can check which colour is what. Here's a map between htop and top mnemonics:

• low == ni
• normal == us
• kernel == sy
• irq == hi
• soft-irq == si
• io-wait == wa
• steal == st
• guest is not shown in top