0

I set up a cronjob on my Raspberry Pi 3B+ (1GB RAM) to reboot every morning, which worked fine for over a year. For three days, however, the RPi has been crashing immediately after reboot.

By “crashing” I mean “freezing” or “becoming unresponsive” – I don't know whether it still runs at all, since it won't react to anything I do (no SSH, all services down etc.). My work-around is to plug out the power cord and plug it in again. After that, the RPi boots (seemingly) fine, at least all services work again and I can log in via SSH.

What might cause this issue and how could I possibly fix it? Thank you!

Debug info

pi@raspberrypi: ~$ uname -a
Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

pi@raspberrypi: ~$ df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root        29G     13G   16G   46% /
devtmpfs        325M       0  325M    0% /dev
tmpfs           455M       0  455M    0% /dev/shm
tmpfs           182M    3,4M  179M    2% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
/dev/mmcblk0p1  255M     31M  225M   13% /boot
tmpfs            91M     20K   91M    1% /run/user/1000

pi@raspberrypi: ~$ crontab -e
@reboot nohup /home/pi/snowflake/proxy/proxy > /home/pi/snowflake/proxy/snowflake.log 2>&1 &
@reboot telegraf --config /etc/telegraf/telegraf.conf

pi@raspberrypi: ~$ sudo crontab -e
40 4 * * * sudo reboot

When setting up my RPi, I had enabled unattended upgrades like this:

sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
sudo systemctl status unattended-upgrades.service

I had also configured the RPi in headless mode:

enter image description here

Here's a list of services that I run on my RPi (including since when I've been hosting them):

  • AdGuard Home (for over a year)
  • Snowflake (for a year)
  • InfluxDB (for a couple of weeks)
  • Grafana (since few days after installing InfluxDB)

After setting up InfluxDB, I configured a USB stick as the swap memory:

pi@raspberrypi: ~$ sudo nano /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=0f70ec87-01  /boot           vfat    defaults          0       2
PARTUUID=0f70ec87-02  /               ext4    defaults,noatime  0       1
UUID=3cfb8767-d24c-48bd-86ad-8942793b5115 none swap sw 0 0
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

Grafana charts

You can clearly see periods when no temperature stats were collected (InfluxDB was down due to the crash, just like the entire system):

enter image description here

The same for the CPU usage chart:

enter image description here

/var/log files

I couldn't paste all of the log files here because they would've exceeded the maximum character count of 30,000 chars, so I uploaded

  • /var/log/syslog
  • /var/log/daemon.log
  • /var/log/kern.log
  • /var/log/messages

to my Nextcloud (no expiration date set): https://murena.io/s/3DAKSyRqiLgADD8

Sorry for the inconvenience.

/var/log/AdGuardHome.err

Here's an excerpt from the AdGuard Home log file from around the time when the RPi crashed. As you can see, there is nothing noteworthy, just an interruption that lasted until I manually rebooted the system:

2023/08/14 00:10:52.307917 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 00:10:55.335954 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 00:16:27.504667 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 00:16:30.503943 [error] dnsproxy: unpacking udp packet: dns: buffer size too small
2023/08/14 04:40:01.707986 [info] service: action run has been done successfully on linux-systemd
2023/08/14 04:40:01.709148 [info] Received signal "terminated"
2023/08/14 04:40:01.709519 [info] stopping AdGuard Home
2023/08/14 11:21:23 [info] AdGuard Home, version v0.108.0-b.43
2023/08/14 11:21:23 [info] service: control action: run
2023/08/14 11:21:23.477455 [info] AdGuard Home, version v0.108.0-b.43
2023/08/14 11:21:23.477565 [info] AdGuard Home is running as a service
2023/08/14 11:21:23.492046 [info] tls: using default ciphers
2023/08/14 11:21:23.502639 [info] safesearch default: disabled
2023/08/14 11:21:23.638269 [info] Initializing auth module: /opt/AdGuardHome/data/sessions.db
2023/08/14 11:21:23.699760 [info] auth: initialized.  users:1  sessions:13
2023/08/14 11:21:23.749410 [info] tls: number of certs: 1
2023/08/14 11:21:29.471087 [info] web: initializing
2023/08/14 11:21:30.293886 [info] dnsproxy: cache: enabled, size 4096 b
2023/08/14 11:21:30.294035 [info] dnsproxy: max goroutines is set to 300
2023/08/14 11:21:30.297410 [info] clients: processing addresses
2023/08/14 11:21:30.302670 [info] AdGuard Home is available at the following addresses:
2023/08/14 11:21:30.303138 [info] go to https://raspberry.pi:8088
2023/08/14 11:21:30.304660 [info] go to http://127.0.0.1:8087
2023/08/14 11:21:30.304812 [info] go to http://[::1]:8087

/var/log/grafana/grafana.log

logger=plugins.update.checker t=2023-08-14T04:33:30.697979208+02:00 level=info msg="Update check succeeded" duration=136.504001ms
logger=server t=2023-08-14T04:40:01.883841513+02:00 level=info msg="Shutdown started" reason="System signal: terminated"
logger=tracing t=2023-08-14T04:40:01.890031171+02:00 level=info msg="Closing tracing"
logger=ticker t=2023-08-14T04:40:02.285024744+02:00 level=info msg=stopped last_tick=2023-08-14T04:40:00+02:00
logger=settings t=2023-08-14T11:21:09.963289565+02:00 level=info msg="Starting Grafana" version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T19:55:59+02:00

/var/log/debug

Aug 13 21:34:40 raspberrypi php7.4-mbstring: php_invoke mbstring: already enabled for PHP 7.4 cli sapi
Aug 13 21:34:41 raspberrypi php7.4-mbstring: php_invoke mbstring: already enabled for PHP 7.4 apache2 sapi
Aug 14 04:40:13 raspberrypi kernel: [    0.000000] pcpu-alloc: s78504 r8192 d32088 u118784 alloc=29*4096
Aug 14 04:40:13 raspberrypi kernel: [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
Aug 14 04:40:13 raspberrypi kernel: [    3.096640] dwc_otg: Microframe scheduler enabled
Aug 14 04:40:13 raspberrypi kernel: [    3.124313] dwc_otg: FIQ enabled
Aug 14 04:40:13 raspberrypi kernel: [    3.124332] dwc_otg: NAK holdoff enabled
Aug 14 04:40:13 raspberrypi kernel: [    3.124348] dwc_otg: FIQ split-transaction FSM enabled
Aug 14 04:40:13 raspberrypi kernel: [    3.124377] Module dwc_common_port init
Aug 14 04:40:13 raspberrypi kernel: [    3.438860]   with arguments:
Aug 14 04:40:13 raspberrypi kernel: [    3.438876]     /sbin/init
Aug 14 04:40:13 raspberrypi kernel: [    3.438892]   with environment:
Aug 14 04:40:13 raspberrypi kernel: [    3.438906]     HOME=/
Aug 14 04:40:13 raspberrypi kernel: [    3.438921]     TERM=linux
Aug 14 04:40:13 raspberrypi kernel: [    5.484127] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
Aug 14 04:40:13 raspberrypi kernel: [   10.822844] brcmfmac: F1 signature read @0x18000000=0x15264345
Aug 14 04:40:13 raspberrypi thd[398]: Found socket passed from systemd
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Successfully called chroot.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Successfully dropped privileges.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Successfully limited resources.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Running.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Watchdog thread running.
Aug 14 04:40:25 raspberrypi rtkit-daemon[848]: Canary thread running.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 1 threads of 1 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:40:26 raspberrypi rtkit-daemon[848]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 4 threads of 2 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:40:27 raspberrypi rtkit-daemon[848]: Supervising 6 threads of 3 processes of 1 users.
Aug 14 04:40:59 raspberrypi kernel: [    0.000000] pcpu-alloc: s78504 r8192 d32088 u118784 alloc=29*4096
Aug 14 04:40:59 raspberrypi kernel: [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
Aug 14 04:40:59 raspberrypi kernel: [    3.096779] dwc_otg: Microframe scheduler enabled
Aug 14 04:40:59 raspberrypi kernel: [    3.124486] dwc_otg: FIQ enabled
Aug 14 04:40:59 raspberrypi kernel: [    3.124507] dwc_otg: NAK holdoff enabled
Aug 14 04:40:59 raspberrypi kernel: [    3.124523] dwc_otg: FIQ split-transaction FSM enabled
Aug 14 04:40:59 raspberrypi kernel: [    3.124553] Module dwc_common_port init
Aug 14 04:40:59 raspberrypi kernel: [    3.777384]   with arguments:
Aug 14 04:40:59 raspberrypi kernel: [    3.777400]     /sbin/init
Aug 14 04:40:59 raspberrypi kernel: [    3.777416]   with environment:
Aug 14 04:40:59 raspberrypi kernel: [    3.777430]     HOME=/
Aug 14 04:40:59 raspberrypi kernel: [    3.777445]     TERM=linux
Aug 14 04:40:59 raspberrypi kernel: [    5.484224] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
Aug 14 04:40:59 raspberrypi kernel: [   10.998883] brcmfmac: F1 signature read @0x18000000=0x15264345
Aug 14 04:40:59 raspberrypi thd[393]: Found socket passed from systemd
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Successfully called chroot.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Successfully dropped privileges.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Successfully limited resources.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Running.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Watchdog thread running.
Aug 14 04:41:11 raspberrypi rtkit-daemon[875]: Canary thread running.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 1 threads of 1 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 2 threads of 2 processes of 1 users.
Aug 14 04:41:12 raspberrypi rtkit-daemon[875]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 3 threads of 2 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 4 threads of 2 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 5 threads of 3 processes of 1 users.
Aug 14 04:41:13 raspberrypi rtkit-daemon[875]: Supervising 6 threads of 3 processes of 1 users.
Aug 14 11:21:15 raspberrypi kernel: [    0.000000] pcpu-alloc: s78504 r8192 d32088 u118784 alloc=29*4096
Aug 14 11:21:15 raspberrypi kernel: [    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
Aug 14 11:21:15 raspberrypi thd[396]: Found socket passed from systemd
...

/var/log/user.log

Aug 14 04:40:01 raspberrypi grafana[1677]: logger=server t=2023-08-14T04:40:01.883841513+02:00 level=info msg="Shutdown started" reason="System signal: terminated"
Aug 14 04:40:01 raspberrypi grafana[1677]: logger=tracing t=2023-08-14T04:40:01.890031171+02:00 level=info msg="Closing tracing"
Aug 14 04:40:02 raspberrypi grafana[1677]: logger=ticker t=2023-08-14T04:40:02.285024744+02:00 level=info msg=stopped last_tick=2023-08-14T04:40:00+02:00
Aug 14 04:40:13 raspberrypi mtp-probe: checking bus 1, device 4: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3"
Aug 14 04:40:13 raspberrypi mtp-probe: bus: 1, device: 4 was not an MTP device
Aug 14 04:40:13 raspberrypi mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.1"
Aug 14 04:40:13 raspberrypi mtp-probe: bus: 1, device: 5 was not an MTP device
Aug 14 04:40:19 raspberrypi vncserver-x11[585,root]: ServerManager: Server started
Aug 14 04:40:20 raspberrypi vncserver-x11[585,root]: ConsoleDisplay: Cannot find a running X server on vt1
Aug 14 04:40:27 raspberrypi pipewire[840]: Failed to receive portal pid: org.freedesktop.DBus.Error.NameHasNoOwner: Could not get PID of name 'org.freedesktop.portal.Desktop': no such name
Aug 14 04:40:30 raspberrypi pulseaudio[841]: Failed to find a working profile.
Aug 14 04:40:30 raspberrypi pulseaudio[841]: Failed to load module "module-alsa-card" (argument: "device_id="1" name="platform-3f902000.hdmi" card_name="alsa_card.platform-3f902000.hdmi" namereg_fail=false tsched=no fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes avoid_resampling=no card_properties="module-udev-detect.discovered=1""): initialization failed.
Aug 14 04:40:59 raspberrypi mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.1"
Aug 14 04:40:59 raspberrypi mtp-probe: checking bus 1, device 4: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3"
Aug 14 04:40:59 raspberrypi mtp-probe: bus: 1, device: 4 was not an MTP device
Aug 14 04:40:59 raspberrypi mtp-probe: bus: 1, device: 5 was not an MTP device
Aug 14 04:41:05 raspberrypi vncserver-x11[578,root]: ServerManager: Server started
Aug 14 04:41:06 raspberrypi vncserver-x11[578,root]: ConsoleDisplay: Cannot find a running X server on vt1
Aug 14 04:41:13 raspberrypi pipewire[867]: Failed to receive portal pid: org.freedesktop.DBus.Error.NameHasNoOwner: Could not get PID of name 'org.freedesktop.portal.Desktop': no such name
Aug 14 04:41:16 raspberrypi pulseaudio[868]: Failed to find a working profile.
Aug 14 04:41:16 raspberrypi pulseaudio[868]: Failed to load module "module-alsa-card" (argument: "device_id="1" name="platform-3f902000.hdmi" card_name="alsa_card.platform-3f902000.hdmi" namereg_fail=false tsched=no fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes avoid_resampling=no card_properties="module-udev-detect.discovered=1""): initialization failed.
Aug 14 04:41:17 raspberrypi pulseaudio[868]: Found duplicated D-Bus path for adapter /org/bluez/hci0
Aug 14 11:21:15 raspberrypi mtp-probe: checking bus 1, device 4: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.3"
Aug 14 11:21:15 raspberrypi mtp-probe: checking bus 1, device 5: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/1-1.1.1"
Aug 14 11:21:15 raspberrypi mtp-probe: bus: 1, device: 4 was not an MTP device
Aug 14 11:21:15 raspberrypi mtp-probe: bus: 1, device: 5 was not an MTP device
...

/var/log/unattended-upgrades/unattended-upgrades-shutdown.log

2023-08-01 04:40:03,477 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-02 04:40:02,370 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-02 21:23:12,174 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-02 21:35:45,862 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-03 00:46:23,362 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-03 04:40:03,563 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-04 04:40:02,322 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-05 04:40:02,769 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-06 04:40:04,135 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-07 04:40:02,830 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-08 04:40:02,940 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-09 04:40:04,721 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-10 04:40:03,076 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-11 04:40:02,682 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-12 04:40:02,736 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-13 04:40:03,958 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-13 12:51:16,042 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running
2023-08-14 04:40:03,959 WARNING - SIGTERM or SIGHUP received, stopping unattended-upgrades only if it is running

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.