0

I currently have a Pi4 4GB running as my homelab but recently I've been facing random crashes that I've been really struggling to diagnose. I run the Pi headless with access over SSH and I'll know when it fails as I can't SSH in, all my web services just fail, any scheduled scripts fail and plugging in a monitor gives me a blank screen. A quick power cycle brings everything back up. It could fail overnight or mid day, no noticeable pattern.

I run more or less everything via Docker on standard Raspbian so have about 18 containers running on the Pi ranging from media stuff to home automation tools. It sounds like a lot to run all on the one Pi but I do have a few monitoring tools running which report an average of 15% CPU usage (with brief highs of 50-80%) and only 1.5GB of RAM usage. CPU temps average at 50 with a fanshim kicking in at 60 so it never breaches that.

I originally thought it was the SD card slowly corrupting itself so I moved to a fresh install (and boot) on an SSD a little while back so it's not that. I also recently swapped my 2A power supply to the official Pi 4 PSU so it shouldn't be that either. I do have the SSD and another HDD attached but they're running via an externally powered USB 3 hub so shouldn't be issues there either.

I've tried searching through /var/log/messages, /var/log/syslog and /var/log/kern.log but there's nothing that jumps out to me as problematic but it's not really my area of expertise either.

This looks concerning in the messages log.... the logs are empty throughout the day and then suddenly at Feb 10 23:17:06 the Pi looks like it randomly tries to reboot itself. I can't find anything to point to what causes this and it looks like it doesn't actually restart properly either.

Feb 10 09:55:01 raspberrypi kernel: [141571.753032] br-0d1467dad996: port 1(vethea97a6c) entered forwarding state
Feb 10 23:17:06 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
Feb 10 23:17:06 raspberrypi kernel: [    0.000000] Linux version 5.4.83-v7l+ (dom@buildbot) (gcc version 8.4.0 (Ubuntu/Linaro 8.4.0-3ubuntu1)) #1379 SMP Mon Dec 14 13:11:54 GMT 2020

Full log pastebin: https://pastebin.com/cEvniTwt

At the moment I have another Pi running HomeAssistant which monitors the Pi4 for downtime and notifies me + toggles a smartplug if there is. Obviously not ideal so I'll move to a NUC if this isn't solvable.

Improve this question
2
  • "but recently". So remove what you changed recently and put it back in stages to identify the fault. I don't really see this is an answerable question or one which is suitable for this site.
    – joan
    Feb 11, 2021 at 10:36
  • Nothing has changed that I can imagine would be causing system crashes from a software side of things, as I mentioned everything runs in docker. I rarely change the host apart from standard updates here and there. I guess I'm looking for some pointers on where to look from people with more sysadmin experience than myself. Helpful comments only please!
    – burg93
    Feb 11, 2021 at 11:07

1 Answer 1

Reset to default
2

One way to find the problematic piece of software is to install them one by one until you face the problem (or uninstall one by one until the problem goes away). If even a vanilla system crashes, the problem is in the hardware.

Other than that, you may want to read this, but it probably won't help much. If there's nothing in the logs, it may be very hard to find the reason.

Improve this answer
6
  • Interesting.... I left a monitor attached and logged in this time (normally fully headless). Came back to it earlier after it collapsed again to a screen full of (and still going) messages "EXT4-fs error"'s. Guessing theres a problem reading the drive out of nowhere sometimes. I've moved the SSD from the powered hub directly into the Pi USB port now to see if that helps. I'll also schedule a fsck on the next startup. I guess this also shows why the logs aren't being written to..... there's no drive to write them to.
    – burg93
    Feb 11, 2021 at 15:51
  • @burg93 That could absolutely be the reason. Linux tends to make filesystems read-only on the first sight of corruption, that's why the error messages never make it to the log. This could be caused by something as simple as an unclean shutdown.
    – Dmitry Grigoryev
    Feb 11, 2021 at 15:58
  • Although I did just read if the file system goes into read only due to corruption fdisk is automatically ran on the next startup. So I'd assume that even without me scheduling one, if I did have an unclean shutdown it should sort itself without intervention next boot. Especially since I have fsck.repair=yes on the cmdline.txt
    – burg93
    Feb 11, 2021 at 16:13
  • @burg93 fdisk is never run automatically. fsck may run on startup, depending on the settings and filesystem state. You can run tune2fs -l /dev/mmcblk0p2 to see when it was last checked.
    – Dmitry Grigoryev
    Feb 11, 2021 at 16:21
  • Sorry my bad on the spelling there, I meant fsck! I've had a bigger look into my SSD boot drive and looks as though there's two partitions on there, the boot one which is FAT32 and then a much larger Linux one..... Could this be causing any issues. tune2fs reports no recent runs on sda2 but last was success but can't run on the FAT32 formatted boot one... /dev/sda1 8192 532479 524288 256M c W95 FAT32 (LBA) /dev/sda2 532480 468862127 468329648 223.3G 83 Linux
    – burg93
    Feb 11, 2021 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.