Headless Raspbian Jessie 8.0 keeps crashing days apart

Question

I leave my raspberry pi on almost all of the time and hardly ever see it crash, but sometimes, maybe every other day or so, I find that I am unable to SSH into my Pi. I go downstairs and see that the green indicator light is no longer flashing, leaving just a solid red.

This has happened many times before and I have already downloaded a fresh IMG file and reflashed the SD card with a fresh install of Jessie, but this keeps happening.

The last entry I can find in the /var/syslog file before the crash is this:

Sep 21 08:17:11 nalusis kernel: [   15.540726] random: nonblocking pool is initialized

The only other thing I can think of is that perhaps the SD card has gone bad, or worse: the Pi board.

Here is a little before the crash:

Sep 21 16:17:06 nalusis systemd[1]: Started Getty on tty1.
Sep 21 16:17:06 nalusis systemd[1]: Starting Serial Getty on ttyAMA0...
Sep 21 16:17:06 nalusis systemd[1]: Started Serial Getty on ttyAMA0.
Sep 21 16:17:06 nalusis systemd[1]: Starting Login Prompts.
Sep 21 16:17:06 nalusis systemd[1]: Reached target Login Prompts.
Sep 21 16:17:08 nalusis apache2[453]: Starting web server: apache2.
Sep 21 16:17:08 nalusis systemd[1]: Started LSB: Apache2 web server.
Sep 21 16:17:08 nalusis systemd[1]: Starting Multi-User System.
Sep 21 16:17:08 nalusis systemd[1]: Reached target Multi-User System.
Sep 21 16:17:08 nalusis systemd[1]: Starting Update UTMP about System Runlevel Changes...
Sep 21 16:17:08 nalusis systemd[1]: Started Update UTMP about System Runlevel Changes.
Sep 21 16:17:08 nalusis systemd[1]: Startup finished in 3.133s (kernel) + 8.767s (userspace) = 11.900s.
Sep 21 16:17:09 nalusis ntpd[474]: Listen normally on 5 eth0 fe80::43a9:4f38:d61d:45f4 UDP 123
Sep 21 16:17:09 nalusis ntpd[474]: peers refreshed
Sep 21 16:17:11 nalusis kernel: [   15.237648] random: nonblocking pool is initialized
...
Had to reboot after

Answer 1

Given the information provided, it's pretty much impossible to tell what's wrong. There are 3 things I'd suggest, in order:

1. Try flashing a new SD card and running from it.

2. Try running a vanilla system with no extra software installed, and without any extra hardware connected to the board (USB sticks and the like). Perhaps you're doing that already, in which case I have bad news for you: you probably do have a faulty RPi.

3. Kernel panic events often don't go to syslog: writing anything to the filesystem is foolish when the system is about to hang up. I wouldn't trust SSH to get all the messages either. If you really want to know what is happening, you should set up a serial UART connection, add console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 to your /boot/cmdline.txt and wait for the crash to happen with another computer collecting messages from the UART. Once you get those messages, check them for clues.

In case you come to step 3, even if you get to know what exactly is wrong, there will probably be no fix for it. Modern electronics are designed to be replaced, not repaired.

Answer 2

Not having the green ACT light flashing does not mean anything is wrong, it just means not much is happening, at least in terms of I/O to the SD card, which is what that light literally indicates -- i.e., it's possible for the system to be very busy and there to be no ACT light activity. It's mostly useful at boot, and for checking that the SD card isn't active if you have to pull the plug, since doing so at that point is more likely to cause corruption.

This is interesting though:

Sep 21 08:17:11 REDACTED kernel: [   15.540726] random: nonblocking pool is initialized

You actually had me searching though logs to find a similar example of a REDACTED message, lol, until I realized you just meant the hostname.

Anyway, that one's a pretty standard one post boot, and the 15.540726 is a timestamp in seconds since the kernel started counting.

In other words, this is from about 15 seconds after a reboot, and other messages leading up to it should confirm that. If you didn't trigger this, you should look backward through the log to before the last 0.000000 message (there are a surprising number of things that happen in the first microsecond), which should be a note about the logger starting up, at which point it dumps these cached messages from the kernel (they don't actually appear in the log in the first microsecond, but by 15 s they should be).

Any messages from right before are from before the system shut down and that may include a clue about about what triggered the reboot. Even if it appears innocuous, anything from 8:10 or 8:15 onward is worth scrutiny.