Using two Pi's to reset each other on failure

Question

I have a Pi Zero W that I'm going to trust to manage some expensive equipment. But two nights in a row, it has failed. It stops executing anything including my script and the ssh daemon so that I have to power cycle it. This is unacceptable reliability, and I can't imagine what my script might be able to do to break the kernel.

So, in absence of reliable hardware, I'm thinking about putting two Pi's together, where through GPIO pins they communicate a heart beat, and if either stops signaling, the other can pull down the RUN pin on the halted one to reset it. This would be a pain to set up, particularly if it won't be adequate to connect one Pi's RUN pin to the other Pi's GND. If I literally have to connect a RUN pin to a GND on the same Pi, then I guess I need a relay, which is a whole nuther bother.

Has anyone run into these problems and can give any advice?

Answer 1

I might be off here but seriously speaking you are not looking for solution why Pi Zero stops working but rather how to detect it. That is not really where I would go in the first place. I have few Zeros that run weeks without any failure - if I was you I would have started discovering WHY it hangs or so. On top of that, if you start your services via systemd, it has the option to restart the process when it fails - see example from my system:

[Unit]
Description=UPS handler
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/python3 /home/pi/scripts/ups-final.py

[Install]
WantedBy=multi-user.target

Answer 2

The Pi SOC has an inbuilt watchdog timer which can be used to manage failures.

Any other system just adds (needless) complexity and testing is a challenge.

Having said this I have never had an unprovoked failure, even though I have several Pi running continuously. They seem to survive through the inevitable power failures.

IMO the Pi Zero is not the best choice; the build quality is poor and they have critical components extending past the edge of the board.

In telecommunications plant it is usual to have parallel processors which can be hot swapped.

Edit

---

There is a BCM hardware watchdog; If you want to start the hardware watchdog include dtparam=watchdog=on in /boot/config.txt

In and of itself this does little, although it should restart the system if not "kicked" regularly. You can write code which opens /dev/watchdog to kick it off.

There is also a watchdog daemon which you can configure to activate the watchdog; you should be able to start with
sudo systemctl enable watchdog

systemctl status watchdog.service will show its status.

You should read man watchdog and it is possible to configure it in /etc/watchdog.conf See man watchdog.conf

I set mine up years ago, and while I did some testing at the time and provoked outages, I just leave it running on all my Pi. I haven't touched it in years, except to debug the systemd service in Jessie.

I believe there is a newer command sudo wdctl /dev/watchdog to check status.

In the "good old days" (1970/1980s) it was common to use an external timer (often a 555) as a watchdog, but this is rarely necessary with modern hardware.