I set up a cronjob on my Raspberry Pi 3B+ (1GB RAM) to reboot every morning, which worked fine for over a year. For three days, however, the RPi has been crashing immediately after reboot.
By “crashing” I mean “freezing” or “becoming unresponsive” – I don't know whether it still runs at all, since it won't react to anything I do (no SSH, all services down etc.). My work-around is to plug out the power cord and plug it in again. After that, the RPi boots (seemingly) fine, at least all services work again and I can log in via SSH.
What might cause this issue and how could I possibly fix it? Thank you!
Debug info
pi@raspberrypi: ~$ uname -a
Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
pi@raspberrypi: ~$ df -h
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root 29G 13G 16G 46% /
devtmpfs 325M 0 325M 0% /dev
tmpfs 455M 0 455M 0% /dev/shm
tmpfs 182M 3,4M 179M 2% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
/dev/mmcblk0p1 255M 31M 225M 13% /boot
tmpfs 91M 20K 91M 1% /run/user/1000
pi@raspberrypi: ~$ crontab -e
@reboot nohup /home/pi/snowflake/proxy/proxy > /home/pi/snowflake/proxy/snowflake.log 2>&1 &
@reboot telegraf --config /etc/telegraf/telegraf.conf
pi@raspberrypi: ~$ sudo crontab -e
40 4 * * * sudo reboot
When setting up my RPi, I had enabled unattended upgrades like this:
sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
sudo systemctl status unattended-upgrades.service
I had also configured the RPi in headless mode:
Here's a list of services that I run on my RPi (including since when I've been hosting them):
- AdGuard Home (for over a year)
- Snowflake (for a year)
- InfluxDB (for a couple of weeks)
- Grafana (since few days after installing InfluxDB)
After setting up InfluxDB, I configured a USB stick as the swap memory:
pi@raspberrypi: ~$ sudo nano /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=0f70ec87-01 /boot vfat defaults 0 2
PARTUUID=0f70ec87-02 / ext4 defaults,noatime 0 1
UUID=3cfb8767-d24c-48bd-86ad-8942793b5115 none swap sw 0 0
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
Grafana charts
You can clearly see periods when no temperature stats were collected (InfluxDB was down due to the crash, just like the entire system):
The same for the CPU usage chart:
/var/log files
I couldn't paste all of the log files here because they would've exceeded the maximum character count of 30,000 chars, so I uploaded
/var/log/syslog/var/log/daemon.log/var/log/kern.log/var/log/messages
to my Nextcloud (no expiration date set): https://murena.io/s/3DAKSyRqiLgADD8
Sorry for the inconvenience.
/var/log/AdGuardHome.err
Here's an excerpt from the AdGuard Home log file from around the time when the RPi crashed. As you can see, there is nothing noteworthy, just an interruption that lasted until I manually rebooted the system:
2023/08/14 00:10:52.307917 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 00:10:55.335954 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 00:16:27.504667 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 00:16:30.503943 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 04:40:01.707986 [info] service: action run has been done successfully on linux-systemd
2023/08/14 04:40:01.709148 [info] Received signal "terminated"
2023/08/14 04:40:01.709519 [info] stopping AdGuard Home
2023/08/14 11:21:23 [info] AdGuard Home, version v0.108.0-b.43
2023/08/14 11:21:23 [info] service: control action: run
2023/08/14 11:21:23.477455 [info] AdGuard Home, version v0.108.0-b.43
2023/08/14 11:21:23.477565 [info] AdGuard Home is running as a service
2023/08/14 11:21:23.492046 [info] tls: using default ciphers
2023/08/14 11:21:23.502639 [info] safesearch default: disabled
2023/08/14 11:21:23.638269 [info] Initializing auth module: /opt/AdGuardHome/data/sessions.db
2023/08/14 11:21:23.699760 [info] auth: initialized. users:1 sessions:13
2023/08/14 11:21:23.749410 [info] tls: number of certs: 1
2023/08/14 11:21:29.471087 [info] web: initializing
2023/08/14 11:21:30.293886 [info] dnsproxy: cache: enabled, size 4096 b
2023/08/14 11:21:30.294035 [info] dnsproxy: max goroutines is set to 300
2023/08/14 11:21:30.297410 [info] clients: processing addresses
2023/08/14 11:21:30.302670 [info] AdGuard Home is available at the following addresses:
2023/08/14 11:21:30.303138 [info] go to https://raspberry.pi:8088
2023/08/14 11:21:30.304660 [info] go to http://127.0.0.1:8087
2023/08/14 11:21:30.304812 [info] go to http://[::1]:8087
/var/log/grafana/grafana.log
logger=plugins.update.checker t=2023-08-14T04:33:30.697979208+02:00 level=info msg="Update check succeeded" duration=136.504001ms
logger=server t=2023-08-14T04:40:01.883841513+02:00 level=info msg="Shutdown started" reason="System signal: terminated"
logger=tracing t=2023-08-14T04:40:01.890031171+02:00 level=info msg="Closing tracing"
logger=ticker t=2023-08-14T04:40:02.285024744+02:00 level=info msg=stopped last_tick=2023-08-14T04:40:00+02:00
logger=settings t=2023-08-14T11:21:09.963289565+02:00 level=info msg="Starting Grafana" version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T19:55:59+02:00
/var/log/debug
Aug 13 21:34:40 raspberrypi php7.4-mbstring: php_invoke mbstring: already enabled for PHP 7.4 cli sapi
Aug 13 21:34:41 raspberrypi php7.4-mbstring: php_invoke mbstring: already enabled for PHP 7.4 apache2 sapi
Aug 14 04:40:13 raspberrypi kernel: [ 0.000000] pcpu-alloc: s78504 r8192 d32088 u118784 alloc=29*4096
Aug 14 04:40:13 raspberrypi kernel: [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Aug 14 04:40:13 raspberrypi kernel: [ 3.096640] dwc_otg: Microframe scheduler enabled
Aug 14 04:40:13 raspberrypi kernel: [ 3.124313] dwc_otg: FIQ enabled
Aug 14 04:40:13 raspberrypi kernel: [ 3.124332] dwc_otg: NAK holdoff enabled
Aug 14 04:40:13 raspberrypi kernel: [ 3.124348] dwc_otg: FIQ split-transaction FSM enabled
Aug 14 04:40:13 raspberrypi kernel: [ 3.124377] Module dwc_common_port init
Aug 14 04:40:13 raspberrypi kernel: [ 3.438860] with arguments:
Aug 14 04:40:13 raspberrypi kernel: [ 3.438876] /sbin/init
Aug 14 04:40:13 raspberrypi kernel: [ 3.438892] with environment:
Aug 14 04:40:13 raspberrypi kernel: [ 3.438906] HOME=/
Aug 14 04:40:13 raspberrypi kernel: [ 3.438921] TERM=linux
Aug 14 04:40:13 raspberrypi kernel: [ 5.484127] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
Aug 14 04:40:13 raspberrypi kernel: [ 10.822844] brcmfmac: F1 signature read @0x18000000=0x15264345
Aug 14 04:40:13 raspberrypi thd[398]: Found socket passed from systemd
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Successfully called chroot.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Successfully dropped privileges.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Successfully limited resources.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Running.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Watchdog thread running.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Canary thread running.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 1 threads of 1 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 4 threads of 2 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 6 threads of 3 processes of 1 users.
Aug 14 04:40:59 raspberrypi kernel: [ 0.000000] pcpu-alloc: s78504 r8192 d32088 u118784 alloc=29*4096
Aug 14 04:40:59 raspberrypi kernel: [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Aug 14 04:40:59 raspberrypi kernel: [ 3.096779] dwc_otg: Microframe scheduler enabled
Aug 14 04:40:59 raspberrypi kernel: [ 3.124486] dwc_otg: FIQ enabled
Aug 14 04:40:59 raspberrypi kernel: [ 3.124507] dwc_otg: NAK holdoff enabled
Aug 14 04:40:59 raspberrypi kernel: [ 3.124523] dwc_otg: FIQ split-transaction FSM enabled
Aug 14 04:40:59 raspberrypi kernel: [ 3.124553] Module dwc_common_port init
Aug 14 04:40:59 raspberrypi kernel: [ 3.777384] with arguments:
Aug 14 04:40:59 raspberrypi kernel: [ 3.777400] /sbin/init
Aug 14 04:40:59 raspberrypi kernel: [ 3.777416] with environment:
Aug 14 04:40:59 raspberrypi kernel: [ 3.777430] HOME=/
Aug 14 04:40:59 raspberrypi kernel: [ 3.777445] TERM=linux
Aug 14 04:40:59 raspberrypi kernel: [ 5.484224] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
Aug 14 04:40:59 raspberrypi kernel: [ 10.998883] brcmfmac: F1 signature read @0x18000000=0x15264345
Aug 14 04:40:59 raspberrypi thd[393]: Found socket passed from systemd
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Successfully called chroot.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Successfully dropped privileges.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Successfully limited resources.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Running.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Watchdog thread running.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Canary thread running.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 1 threads of 1 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 4 threads of 2 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 6 threads of 3 processes of 1 users.
Aug 14 11:21:15 raspberrypi kernel: [ 0.000000] pcpu-alloc: s78504 r8192 d32088 u118784 alloc=29*4096
Aug 14 11:21:15 raspberrypi kernel: [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
Aug 14 11:21:15 raspberrypi thd[396]: Found socket passed from systemd
...
/var/log/user.log
Aug 14 04:40:01 raspberrypi grafana[1677]: logger=server t=2023-08-14T04:40:01.883841513+02:00 level=info msg="Shutdown started" reason="System signal: terminated"
Aug 14 04:40:01 raspberrypi grafana[1677]: logger=tracing t=2023-08-14T04:40:01.890031171+02:00 level=info msg="Closing tracing"
Aug 14 04:40:02 raspberrypi grafana[1677]: logger=ticker t=2023-08-14T04:40:02.285024744+02:00 level=info msg=stopped last_tick=2023-08-14T04:40:00+02:00
Aug 14 04:40:13 raspberrypi mtp-probe: checking bus 1, device 4: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3"
Aug 14 04:40:13 raspberrypi mtp-probe: bus: 1, device: 4 was not an MTP device
Aug 14 04:40:13 raspberrypi mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.1"
Aug 14 04:40:13 raspberrypi mtp-probe: bus: 1, device: 5 was not an MTP device
Aug 14 04:40:19 raspberrypi vncserver-x11[585,root]: ServerManager: Server started
Aug 14 04:40:20 raspberrypi vncserver-x11[585,root]: ConsoleDisplay: Cannot find a running X server on vt1
Aug 14 04:40:27 raspberrypi pipewire[840]: Failed to receive portal pid: org.freedesktop.DBus.Error.NameHasNoOwner: Could not get PID of name 'org.freedesktop.portal.Desktop': no such name
Aug 14 04:40:30 raspberrypi pulseaudio[841]: Failed to find a working profile.
Aug 14 04:40:30 raspberrypi pulseaudio[841]: Failed to load module "module-alsa-card" (argument: "device_id="1" name="platform-3f902000.hdmi" card_name="alsa_card.platform-3f902000.hdmi" namereg_fail=false tsched=no fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes avoid_resampling=no card_properties="module-udev-detect.discovered=1""): initialization failed.
Aug 14 04:40:59 raspberrypi mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.1"
Aug 14 04:40:59 raspberrypi mtp-probe: checking bus 1, device 4: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3"
Aug 14 04:40:59 raspberrypi mtp-probe: bus: 1, device: 4 was not an MTP device
Aug 14 04:40:59 raspberrypi mtp-probe: bus: 1, device: 5 was not an MTP device
Aug 14 04:41:05 raspberrypi vncserver-x11[578,root]: ServerManager: Server started
Aug 14 04:41:06 raspberrypi vncserver-x11[578,root]: ConsoleDisplay: Cannot find a running X server on vt1
Aug 14 04:41:13 raspberrypi pipewire[867]: Failed to receive portal pid: org.freedesktop.DBus.Error.NameHasNoOwner: Could not get PID of name 'org.freedesktop.portal.Desktop': no such name
Aug 14 04:41:16 raspberrypi pulseaudio[868]: Failed to find a working profile.
Aug 14 04:41:16 raspberrypi pulseaudio[868]: Failed to load module "module-alsa-card" (argument: "device_id="1" name="platform-3f902000.hdmi" card_name="alsa_card.platform-3f902000.hdmi" namereg_fail=false tsched=no fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes avoid_resampling=no card_properties="module-udev-detect.discovered=1""): initialization failed.
Aug 14 04:41:17 raspberrypi pulseaudio[868]: Found duplicated D-Bus path for adapter /org/bluez/hci0
Aug 14 11:21:15 raspberrypi mtp-probe: checking bus 1, device 4: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3"
Aug 14 11:21:15 raspberrypi mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.1"
Aug 14 11:21:15 raspberrypi mtp-probe: bus: 1, device: 4 was not an MTP device
Aug 14 11:21:15 raspberrypi mtp-probe: bus: 1, device: 5 was not an MTP device
...
/var/log/unattended-upgrades/unattended-upgrades-shutdown.log
2023-08-01 04:40:03,477 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-02 04:40:02,370 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-02 21:23:12,174 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-02 21:35:45,862 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-03 00:46:23,362 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-03 04:40:03,563 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-04 04:40:02,322 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-05 04:40:02,769 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-06 04:40:04,135 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-07 04:40:02,830 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-08 04:40:02,940 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-09 04:40:04,721 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-10 04:40:03,076 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-11 04:40:02,682 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-12 04:40:02,736 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-13 04:40:03,958 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-13 12:51:16,042 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-14 04:40:03,959 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
