1

I've been using a raspberry pi 4B with 8GB of ram for about a year now.

Common Features

  • Raspberry Pi 4B 8GB
  • booting from an SSD
  • connected to the internet via wired connection
  • running cron jobs, apache webserver (not exposed to the internet), minidlna
pi@mulberrypi:/var/log $ cat /etc/rpi-issue
Raspberry Pi reference 2021-05-07
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, dcfd74d7d1fa293065ac6d565711e9ff891fe2b8, stage4

pi@mulberrypi:/var/log $ vcgencmd version
Dec  1 2021 15:01:54 
Copyright (c) 2012 Broadcom
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)

pi@mulberrypi:/var/log $ uname -a
Linux mulberrypi 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux

Setup A - No Issues

  • Located near router connected with short cat 5e cable
  • supplying non-internet connected wifi to cameras

Setup B - Strange Crash that impacts router

  • Located further away from router connected with a longer cat6 cable
  • powered usb hub provides usb direct internet connections to the cameras
  • non-internet wifi still running but nothing connected to it

Odd features of the crash

  • It usually crashes in the middle of the night. Last night it crashed at precisely 2:32am. That doesn't match any of the cron jobs I have running and nothing would be running at that point.
  • There are no entries in syslog, messages, kern.log, or any other log files for several minutes near the time of the crash.
  • when it reboots the time stamp is wrong until it connects to ntp (last entry in any log is at 2:32am; more recent reboot at 9:16pm).
  • when crashed the device light is still on but the pi cannot be connected to via the wired connection or its wireless network
  • when it crashes, it somehow takes down the router which supplies it with internet. Disconnecting the pi's ethernet cable fixes the internet.

things I could imagine

  1. usb hub is crashing causing the rpi to lose its usb subsystem and thus not access the OS and not log anything
  2. power supply is having issues and putting the device in a weird state
  3. somehow the rpi is trying to do PoE and killing a router that's not designed to supply that sort of power over the cable?

My questions is basically two-fold:

  1. How can I log this crashing to better identify the cause (since it appears to be leaving no log entries anywhere near there)?
  2. Is there any known issue that would cause this?

Update

the exact times 3:32 and 9:16 suggest that this could be related to cron jobs that pull files off a USB hub and then use ffmpeg to process the files...

the majority of the devices are on a powered USB hub the only exception is the SSD that the pi uses as its boot device

I'm now wondering if there's some sort of access conflict between the two USB3 ports that is causing a crash.

I'm still a bit confused as to why the crash then impacts the router, but that's the best I've got.

8
  • Pi's don't have PoE. That the Pi crashing causes the problem is a presumption. That unplugging the cable "fixes" the router is not evidence of that, only that the connection is propagating a problem from one device to the other. I don't see how this is possible short of defective hardware, either on the Pi or the router. "Last night it crashed at precisely 2:32am" -> If that's based on the time it thinks it is when it reboots (until ntp), it did not crash precisely then, it crashed at some point after that, possibly a number of minutes, FYI (if this matters).
    – goldilocks
    Commented Jun 16, 2022 at 16:25
  • "It clobbers the logs when it crashes -- there's nothing in syslog, dmesg, etc." -> Dmesg is naturally clobbered when the OS shuts down since it is an in memory record. WRT to syslog, are you saying that the log files that are normally in /var/log are gone? Or do you just mean there is nothing in the log to indicate anything about the crash? In that case you should remove the word "clobber" since it is misleading. Corruption at the end of a log (in the form of random bytes) is symptomatic of a power fail.
    – goldilocks
    Commented Jun 16, 2022 at 16:31
  • Great points. Clarifying: There's nothing in the log files near the time of crash; also no gibberish in the log files at that time point. Can narrow the crash time to a minute interval due to a recurring message from a network issue.
    – virmaior
    Commented Jun 16, 2022 at 21:52
  • don't see how this is possible short of defective hardware, either on the Pi or the router. I guess that means that the hardware with a problem could be 1. Pi. 2. Router. 3. Cat 6 cable between them. The devices on the powered usb hub seem to survive with no ill effects.
    – virmaior
    Commented Jun 16, 2022 at 21:56
  • Have you checked for an IP Address conflict? Is the PI static in a DHCP range? Could there be another device/person trying to use that address?
    – RP_Joe
    Commented Jun 17, 2022 at 2:27

1 Answer 1

1

Proving that this can only be because of defective hardware is a matter of demonstrating that it cannot be because of some piece of software has done something it shouldn't do, but could by accident, or because of a hack/bug/what have you.

WRT to something involving PoE, I:

  1. Doubt very much the protocol would be so dunderheaded as to create this potential.
  2. Do not think it is in keeping with how electricity works, presuming that resistors are in use limiting the current on the router eth jack lines.

But I won't bother to elaborate since Pi's do not have onboard PoE cabability anyway.

Moving on, the current state of the internet would seem evidence of the fact that it is not just desirable for it not to be possible for a system to send some signal, maliciously or by accident, to some other system on a network that will seriously foul it's internal state, but by and large this really is the case: It can't be said that it is impossible, but it can be said that it is far from easy and requires:

  1. Carefully crafted maliciousness that targets a specific context.
  2. Some kind of bug that miraculously duplicates #1, which is on the level of "if you left enough monkeys in a room with typewriters long enough, one of them would eventually produce a Shakespeare play" (which is just a horrible abuse of the concept of infinity).

Being directly connected with an ethernet cable may allow for some things that are not possible with just any two nodes on a network, but these are limited to the lowest levels but one of the networking stack. This is significant in that they are also the most fundamental; these aren't things you configure, nor are they pieces of software that some people use on a network. They are, WRT to the linux kernel (which is also what your router likely runs), used constantly by all systems using ethernet.

That there would be a bug that could produce this result and yet leave no trace in the logs, nor already have been found (since it produces such serious consequences), seems very far fetched to me compared to the possibility that:

  1. There's a short/defect in one of the devices that is causing the problem.

Of course, it can always be some sort of malware, but note both devices would have to be cracked -- again it is not simply a matter of software doing something it shouldn't but potentially could in order to affect the other system in a conventional way.

6
  • Everything here makes sense. Do you have any ideas how I can try to have it record logs closer to the moment of the crash on the pi? like can I connect my mac and have it be copying over the log files in real time or something like that?
    – virmaior
    Commented Jun 16, 2022 at 22:58
  • I would suggest that you should replace your CAT-6 cable. This looks to be the main thing changed that is directly connected to the router.
    – NomadMaker
    Commented Jun 16, 2022 at 23:06
  • WRT the logs, I would guess the cause is momentary power loss, and there is no sequence to log that begins with the OS having a premonition that the plug is about to be pulled. BTW, I did not notice the detail about the two cables, I'd assume there's some potential for a short in there, which although long cat cables aren't cheap is still better than the Pi or router having a busted jack. The last strange bit is why it happens around the same time every 24 hours (if it does); ISP routers are prone to update and reboot in the wee hours but not every day.
    – goldilocks
    Commented Jun 17, 2022 at 11:48
  • haven't been able to reproduce this yet. A new hypothesis was overheating, i.e. that the previous location had better airflow, so I've tried to improve that as a first step ... also added some more logging including cpu temperature. If it recurs, next step will involve the cable...
    – virmaior
    Commented Jun 20, 2022 at 10:44
  • @goldilocks are there any known issues with the USB3 controller of RPi4s?
    – virmaior
    Commented Jun 21, 2022 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.