6

I'm running my RPi B in headless mode, controlling it via SSH (Raspbian).

I used to get random crashes under load, until I got a better power supply.

Now, I've found the problem has re-surfaced. I'm trying to compile a new version of Python, and it simply reboots during the make process. It's also crashed on other occasions, especially when it has been under heavy load.

I've removed all peripherals in order to lower the power consumption, but the issue remains.

Any ideas? I'm not sure where to begin investigating an unplanned reboot.

7
  • 1
    sudo grep -i OOM /var/log/syslog.
    – goldilocks
    Jan 25, 2016 at 17:39
  • 2
    What's the rated output of your power supply?
    – Jacobm001
    Jan 25, 2016 at 17:45
  • 2
    @goldilocks a bunch of wd_keepalive[2871]: unable to disable oom handling!. I'll have to try to force a crash again to see if the times of these coincide with the crashes. Btw, what does this mean?
    – elsurudo
    Jan 25, 2016 at 18:41
  • 1
    @Jacobm001 5V max. 2100 mA
    – elsurudo
    Jan 25, 2016 at 18:43
  • That you are getting OutOfMemory messages in your syslog is not promising - it means that the OS thinks there is not enough memory available and it is (desperately?) killing off processes to try to recover. I suspect @goldilocks thinks that this compilation is going to need a lot of memory and wanted you to check. One thought that crosses my mind is that perhaps you have a Quad-core RPi 2B and are trying to compile using all of the cores (with a make -j 5) - reducing that -j value will reduce the number of cores used and memory requirement at the cost of taking longer..!
    – SlySven
    Jan 26, 2016 at 17:05

3 Answers 3

9

a bunch of wd_keepalive[2871]: unable to disable oom handling!. I'll have to try to force a crash again to see if the times of these coincide with the crashes...

wd_keepalive is a watchdog daemon described in man wd_keepalive. I do not think it is installed by default, so you are probably aware of it.

The OOM killer is a kernel feature intended to keep the system up when all available physical memory is consumed. When this happens, the system may become temporarily unresponsive, which may trigger your watchdog. This would explain the spontaneous reboots, since that is what a watchdog usually does.

/var/log/syslog most likely contains a record of the OOM killer activity. This is in the form of a table with columns like this:

[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name

The kernel makes its decision based on some algorithm that generally selects the process with the largest amount of memory consumed, the point being to free up as much memory as possible. However, this does not always stop the "culprit", which may be, for example, something which rapidly forks or balloons when not much was left. It is worth observing that there may not really be any culprit other than the one between chair and keyboard -- everything is behaving as it is supposed to, but you are asking for too much.

At the end of the table you will find the decision made, Out of memory: Kill process 1234 foobar where 1234 is a pid and foobar is the name of the process. Again, this does not necessarily indicate misbehavior, however, if something is misbehaving and that was not it, the OOM condition will probably reoccur quickly and you will find another table further on. By evaluating these you should be able to figure out the details of what happened.

To be clear: The intention of the OOM killer is not to disable the system. It is to free up memory in order to restore normal operation. This is not always effective, and the slow down created by the OOM condition may have triggered the watchdog (which is effective, at least in terms of restarting the system).

To help prevent or diagnose the issue in the future, you could use a monitor such as top or htop. For the long term (e.g., on a server), you could try monitoring specific processes with plog.

9
  • 1
    Yes, I do recall installing this, but honestly it as so long ago I forgot. From the manpage: "Under high system load wd_keepalive might be swapped out of memory and may fail to make it back in in time. Under these circumstances the Linux kernel will reset the machine. To make sure you won't get unnecessary reboots make sure you have the variable realtime set to yes in the configuration file watchdog.conf. This adds real time support to wd_keepalive: it will lock itself into memory and there should be no problem even under the highest of loads." Hmmm...
    – elsurudo
    Jan 25, 2016 at 19:20
  • 1
    Actually, it seems realtime is already set to "yes", so this won't solve the problem
    – elsurudo
    Jan 25, 2016 at 19:21
  • 1
    At a certain point I do not think it will be able to guarantee that, and under such conditions you probably want it to trigger because the system really is unrecoverable -- for example the fork bomb which Jacobm001 mentioned. Fork bombs are easy to create accidently; I believe this recent question is an example of such (picking through the wreckage will confirm that). Obviously make could lead to anything, it depends on the makefile.
    – goldilocks
    Jan 25, 2016 at 19:26
  • 1
    I'm about to disable the watchdog as @Jacobm001 suggested, and see if I can at least make. Not sure what a long-term solution would be though, as I recall this happening during a lot of compiles (which eventually succeeded, given enough tries).
    – elsurudo
    Jan 25, 2016 at 19:28
  • 2
4

wd_keepalive[n] is a thread from your watchdog process. N is an integer specifying the pid of whichever process it's actively watching. From what I can tell, the error is a bug in watchdog. Esentially it's either causing or receiving a fork bomb which then causes the system to crash.

Try disabling watchdog, and then run your make process.

1
  • Disabling the watchdog did indeed allow me to compile, but I don't want to disable it altogether, so it's not a lasting solution
    – elsurudo
    Jan 25, 2016 at 22:28
1

I had a similar problem where the raspberry just crashed when using make. I changed the power adapter and it worked like magic.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.