0

I have a Pi Zero W that I'm going to trust to manage some expensive equipment. But two nights in a row, it has failed. It stops executing anything including my script and the ssh daemon so that I have to power cycle it. This is unacceptable reliability, and I can't imagine what my script might be able to do to break the kernel.

So, in absence of reliable hardware, I'm thinking about putting two Pi's together, where through GPIO pins they communicate a heart beat, and if either stops signaling, the other can pull down the RUN pin on the halted one to reset it. This would be a pain to set up, particularly if it won't be adequate to connect one Pi's RUN pin to the other Pi's GND. If I literally have to connect a RUN pin to a GND on the same Pi, then I guess I need a relay, which is a whole nuther bother.

Has anyone run into these problems and can give any advice?

2
  • I run a pi with sd mounted as read only (logs and changing files are on a nfs) and a nightly cron reboot. hasn't run into any issues yet after a year and change. if you have two, you can have one reboot while the other takes over the task. in my case I could pick up after reboot without missing anything so I only needed one.
    – Abel
    Jun 26, 2021 at 22:06
  • 4
    it may be of value to determine what is causing the issue
    – jsotola
    Jun 26, 2021 at 22:51

2 Answers 2

2

I might be off here but seriously speaking you are not looking for solution why Pi Zero stops working but rather how to detect it. That is not really where I would go in the first place. I have few Zeros that run weeks without any failure - if I was you I would have started discovering WHY it hangs or so. On top of that, if you start your services via systemd, it has the option to restart the process when it fails - see example from my system:

[Unit]
Description=UPS handler
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/python3 /home/pi/scripts/ups-final.py

[Install]
WantedBy=multi-user.target
1
  • I agree completely with you to not fixing a symptom. But your suggestion will not help in this situation. Systemd only checks the starting of a service. If it is running once it does not monitor the service and will not restart it if it get stuck.
    – Ingo
    Jun 29, 2021 at 7:58
2

The Pi SOC has an inbuilt watchdog timer which can be used to manage failures.

Any other system just adds (needless) complexity and testing is a challenge.

Having said this I have never had an unprovoked failure, even though I have several Pi running continuously. They seem to survive through the inevitable power failures.

IMO the Pi Zero is not the best choice; the build quality is poor and they have critical components extending past the edge of the board.

In telecommunications plant it is usual to have parallel processors which can be hot swapped.

Edit


There is a BCM hardware watchdog; If you want to start the hardware watchdog include dtparam=watchdog=on in /boot/config.txt

In and of itself this does little, although it should restart the system if not "kicked" regularly. You can write code which opens /dev/watchdog to kick it off.

There is also a watchdog daemon which you can configure to activate the watchdog; you should be able to start with
sudo systemctl enable watchdog

systemctl status watchdog.service will show its status.

You should read man watchdog and it is possible to configure it in /etc/watchdog.conf See man watchdog.conf

I set mine up years ago, and while I did some testing at the time and provoked outages, I just leave it running on all my Pi. I haven't touched it in years, except to debug the systemd service in Jessie.

I believe there is a newer command sudo wdctl /dev/watchdog to check status.

In the "good old days" (1970/1980s) it was common to use an external timer (often a 555) as a watchdog, but this is rarely necessary with modern hardware.

3
  • Can you tell me more about this "Pi SOC" or share a link? Can I adjust the timer? What conditions does it look for at each interval? I agree, if there's something I can do to self-reset, that would be ideal. I could go with a full Pi 4 I guess. It just seemed overkill for the very simple script it needs to run. Jun 26, 2021 at 22:54
  • Perhaps you're talking about the watchdog that I can install with apt? I found diode.io/raspberry%20pi/… Jun 26, 2021 at 22:56
  • Well, I found and read a few watchdog articles. I managed enough to hork my Pi presumably through an infinite reboot. I haven't figured out how to configure it I guess. Jun 27, 2021 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.