Addressing Point 2
...and how do I troublshoot kernel panics?
A kernel panic is just a crash inside the kernel. A crash caused by any number of normal software or hardware faults.
Debugging the kernel is no different than any other piece of software. Some combination of
- Examining Log Messages
- Examining Stack Traces
- Using a Debugger with Breakpoints
- Fault Isolation (strip/disable software components until only the at-fault section remains running)
One additional option for kernels
- Monitoring kernel exposed internal monitoring structures under
/sys/ , this can help you, for example, track trends (e.g. number of exceptions increases before a crash, CPU load spike, Lots of swapping/context switches). But this is very qualitiative and "not real time" debugging information.
Unfortunately, because the kernel runs the system it is harder to debug in place than user space code. Log Messages are pretty much all you really have fr
It is possible to debug your own kernel code in situ, when you know what it is doing and where it is going wrong using verbose logging and other log-bassed debugging in your custom module/kernel , but diagnosing intermittent crashes in a pre-compiled release kernel is pretty much out of the question. You won't do any better than logging without additional hardware
You need a hardware interface to run the debugging, in the embedded world this is known as
In Circuit Emulation (ICE), and is commonly achieved by using the
Namely, you will need to use
JTAG, which is a hardware debugging interface. This allows one to set breakpoints and interrupt the CPU using external hardware.
When set up correctly, you can use
JTAG easily with
gdb running on a host PC to debug embedded linux kernels. The use is identical to using
gdb with any other application, but the interface is hardware.
You would use this setup
- "catch" (break) these kernel panics before they occur
- The breakpoint will pause the CPU
- Step the CPU through the crash command by command
- Examine all the memory that gets modified/changed
- Examine memory and stack of the CPU using your debugger
- Use this information to determine what is the root cause of the crash
Good Resource Tutorial: https://www.elinux.org/Debugging_The_Linux_Kernel_Using_Gdb
Note that even this may not be sufficient, there are many problems that only occur when things are running "at speed", that is, the interjection of a debugger or even additional log messages may change the system enough to hide or mask the bug.
It's more of an art than a science
Your log is actually truncated. I suspect you have a hardware fault that triggers an unhandled CPU exception that causes the kernel crash/panic.
One very common scenario is intermittent/failing/corrupt memory that causes an incorrect command to be loaded into the CPU which causes an exception.