1

So after some help to locate the problem, I found the problem (I hope). Now I need help to fix it.

My setup:

  • A Raspberry Pi 2 model B
  • Running owncloud.
  • Open to the world. (nginx)
  • Raspbian. Is fully up to date.
  • Has a dedicated power supply which never gave problems with my other Pi's, even tried another power supply.
  • Connected by Lan
  • Has an external hard drive with it's own power supply.
  • Functions quite a bit like a NAS via SFTP.
  • Connected to a TV with a keyboard, for debug purposes.

Sometimes it freezes within hours, this time it was after a few days. This last time I had the HDMI plugged into see a screen full of messages I don't understand. It ends with:

bad: scheduling from idle thread!

And is not responding to anything, ctrl+c nothing works, except pulling the plug and rebooting.

Can anybody help me out?

Bad picture

EDIT

I had an other crash the message here was:

note: nginx[] exited with preempt_count 2

1 Answer 1

3

This is a not good, as in bad:[, sign; it may have to do with a driver, possibly the driver for the Pi's SMSC ethernet controller, probably something more obscure since it likely isn't something that's affected many people.

Those console messages are from the kernel, and that final one only occurs once in the kernel source, in kernel/sched/idle_task.c (bare with me for a bit if this is beginning to sound like gobbledygook). It's been around for at least a decade (probably closer to two) but there are surprisingly few references to it either in the linux kernel mailing list archive or online generally (and literally only a couple associated with "raspberry").

The "idle task" is one that runs when there's nothing else to do. Tasks, including this one, can be pre-empted so the CPU can be used to do something more urgent; they can also make themselves inactive (aka. idle), so something else can happen -- but not this one, because it's already "the idle task". It can't claim it has nothing to do, because it's what's supposed to happen when everything else has already said that. It should just keep doing nothing until pre-empted.

However, a kernelspace programming error could produce a situation where the idle task does claim it's not busy and should be made inactive. Since that is definitely not supposed to happen, the idle task then reports this error and produces a stack trace, which is a list of references to recently executed code leading up the problem. That's all the stuff before the "bad" message". Stack traces lead into the past, so the last thing there is the earliest thing to happen (nb., the message timestamps on the left have nothing to do with that). At the top of the screen there's some references to the TCP/IP networking module, and further up would be whatever actually triggered the error (which is why I'm guessing the ethernet driver, but that is not necessarily so).

I can't say whether this error in itself should be fatal to the system but it could, and is at least related to the chain of events.

Anyway...

Looking at the timestamps on the left (seconds since boot, looks like 3-4 days), that stack trace took at least a second to produce and hopefully it or something relevant that happened right before it got logged. Although the new version of Raspbian (8) uses a new logger (journald) in addition to the old logger (rsyslogd), all versions still use the latter to log everything in plain text to /var/log/syslog, including stuff from the kernel.

Everything in syslog has a dated timestamp; for the kernel stuff this is in addition to the seconds since boot. E.g.,

Nov 14 12:20:38 localhost kernel: [142480.895132]  [<ffffffff811db24a>] __mark_inode_dirty+0x24a/0x310
^^^^^^^^^^^^^^^                    ^^^^^^^^^^^^^

This is also part of a stack trace (noticeable because there's a <memory address> there too); they are not necessarily indications of a fatal error. Point being, yours will be tagged something like that (your hostname may not be localhost) and if you can't find it in syslog then you can at least find the last thing there (also, with a little math you can correspond kernel timestamps to syslog timestamps).

"The last thing there" will be immediately before the messages for the next reboot. The very first one of those will include localhost rsyslogd: [blah blah blah] starting (again your hostname may be different), then a long block of kernel messages tagged [ 0.000000]. On the Pi at that point the syslog timestamp will be Dec 31 since the clock has not been set properly.

11
  • Thanks for the in dept answer. It's a bit technical. But if I understand it the TLDR should be. 1. note the seconds 2. reboot 3. try to find the last thing before the boot in /var/log/syslog 3b. match the time stamp 4. try to find a useful message/error that happend before that. 5, search that. Right?
    – janw
    Commented Nov 24, 2015 at 21:02
  • 1
    Yeah. You don't really have to bother with matching the timestamps if you are looking for something right before boot, since that will just be whatever is immediately before the stuff tagged [ 0.000000] (although beware one log file can hold any number of boots). You could then compare what's there to the stuff from the screenshot to see if it matches up and what else it says; this also gives you something to more easily cut and paste.
    – goldilocks
    Commented Nov 24, 2015 at 21:13
  • Okay. It doesn't crash as much recently. And my Wife unplugs it because she needs it. So I'm waiting for the moment it crashes and I'm the first one to reach it. Thanks for the info.
    – janw
    Commented Nov 25, 2015 at 7:57
  • Those logs will still be there unless you've cleared them out. grep and less are good tools for finding this stuff from the command line, e.g. grep "0.000000]" /var/log syslog | less will show that boot sequence -- but that is not actually what you are looking for. grep -B 25 "localhost rsyslogd:" /var/log/syslog* | less should show you a series of 25 line records from before boot (i.e. shutdown/death), but you might have to change localhost to the right thing.
    – goldilocks
    Commented Nov 25, 2015 at 12:43
  • 1
    Followup. I replaced the raspberry itself with a other one. Same power supply, SD card ect.. It works without problems. So it waas the Pi hardware.
    – janw
    Commented Jan 12, 2016 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.