PXE - reboot if nfs root fail

Question

I have a full working PXE server and multiple Raspberry Pi 3b+ running on read only.

My problem come from an issue we can face anytime, a connection loss.

If we lose connection with the NFS server, the Raspberry Pi will lose its root directory. So it can't run anything and will only react with RAM loaded applications.

We tried a script witch found the connection lost, write a log but can't launch "reboot" command.

We tried to put echo b > /proc/sysrq-trigger but we need and root authorisation and crontab script and so one always ask for password.

We also tried to put a line on rc.local but, the process is killed after a while.

This project is an automatic booting kiosk which got no input at all.

while true; do    if tail /var/log/messages | grep 'server 10.10.52.3 not responding'; then echo b > /proc/sysrq-trigger; fi; done;

We don't want to ping anything or surcharge Raspberry Pi. The monitoring is already heavy and we got GPIO reader too.

Answer 1

You do not say that you use a different (local?) partition for the /var directory. So /var/log/messages is also stored on the root file system that is mounted from the server by NFS. If you want to look at it if the network connection is not responding then it cannot work. If the connection is broken you cannot look at the log file if its broken.

You may have a look at journalctl --boot if you also find the message server 10.10.52.3 not responding. It should do because systemd by default logs also everything in its journal. Also by default the journal is stored in /run/log/journal that is mounted as tmpfs, means in the ram. So you do not have the problem looking at a file that isn't available on error. Have a look at man journald.conf to fine tune your journal (Storage, SystemMaxUse, etc).

But I think the easiest way to check your system is to:

rpi ~$ sudo apt install watchdog

and reboot. With it you have many options to observe the system by just configuring /etc/watchdog.conf. For example you can check an ip address by ping, access a file, network connection, and many other things. Just look at man watchdog.conf.