I currently have a Pi4 4GB running as my homelab but recently I've been facing random crashes that I've been really struggling to diagnose. I run the Pi headless with access over SSH and I'll know when it fails as I can't SSH in, all my web services just fail, any scheduled scripts fail and plugging in a monitor gives me a blank screen. A quick power cycle brings everything back up. It could fail overnight or mid day, no noticeable pattern.
I run more or less everything via Docker on standard Raspbian so have about 18 containers running on the Pi ranging from media stuff to home automation tools. It sounds like a lot to run all on the one Pi but I do have a few monitoring tools running which report an average of 15% CPU usage (with brief highs of 50-80%) and only 1.5GB of RAM usage. CPU temps average at 50 with a fanshim kicking in at 60 so it never breaches that.
I originally thought it was the SD card slowly corrupting itself so I moved to a fresh install (and boot) on an SSD a little while back so it's not that. I also recently swapped my 2A power supply to the official Pi 4 PSU so it shouldn't be that either. I do have the SSD and another HDD attached but they're running via an externally powered USB 3 hub so shouldn't be issues there either.
I've tried searching through /var/log/messages, /var/log/syslog and /var/log/kern.log but there's nothing that jumps out to me as problematic but it's not really my area of expertise either.
This looks concerning in the messages log.... the logs are empty throughout the day and then suddenly at Feb 10 23:17:06 the Pi looks like it randomly tries to reboot itself. I can't find anything to point to what causes this and it looks like it doesn't actually restart properly either.
Feb 10 09:55:01 raspberrypi kernel: [141571.753032] br-0d1467dad996: port 1(vethea97a6c) entered forwarding state Feb 10 23:17:06 raspberrypi kernel: [ 0.000000] Booting Linux on physical CPU 0x0 Feb 10 23:17:06 raspberrypi kernel: [ 0.000000] Linux version 5.4.83-v7l+ (dom@buildbot) (gcc version 8.4.0 (Ubuntu/Linaro 8.4.0-3ubuntu1)) #1379 SMP Mon Dec 14 13:11:54 GMT 2020
Full log pastebin: https://pastebin.com/cEvniTwt
At the moment I have another Pi running HomeAssistant which monitors the Pi4 for downtime and notifies me + toggles a smartplug if there is. Obviously not ideal so I'll move to a NUC if this isn't solvable.