2

I have a full working PXE server and multiple Raspberry Pi 3b+ running on read only.

My problem come from an issue we can face anytime, a connection loss.

If we lose connection with the NFS server, the Raspberry Pi will lose its root directory. So it can't run anything and will only react with RAM loaded applications.

We tried a script witch found the connection lost, write a log but can't launch "reboot" command.

We tried to put echo b > /proc/sysrq-trigger but we need and root authorisation and crontab script and so one always ask for password.

We also tried to put a line on rc.local but, the process is killed after a while.

This project is an automatic booting kiosk which got no input at all.

while true; do    if tail /var/log/messages | grep 'server 10.10.52.3 not responding'; then echo b > /proc/sysrq-trigger; fi; done;

We don't want to ping anything or surcharge Raspberry Pi. The monitoring is already heavy and we got GPIO reader too.

Improve this question
0

You do not say that you use a different (local?) partition for the /var directory. So /var/log/messages is also stored on the root file system that is mounted from the server by NFS. If you want to look at it if the network connection is not responding then it cannot work. If the connection is broken you cannot look at the log file if its broken.

You may have a look at journalctl --boot if you also find the message server 10.10.52.3 not responding. It should do because systemd by default logs also everything in its journal. Also by default the journal is stored in /run/log/journal that is mounted as tmpfs, means in the ram. So you do not have the problem looking at a file that isn't available on error. Have a look at man journald.conf to fine tune your journal (Storage, SystemMaxUse, etc).

But I think the easiest way to check your system is to:

rpi ~$ sudo apt install watchdog

and reboot. With it you have many options to observe the system by just configuring /etc/watchdog.conf. For example you can check an ip address by ping, access a file, network connection, and many other things. Just look at man watchdog.conf.

Improve this answer
4
  • I'm currently looking to make watchdog works but got an issue with /dev/watchdog which is busy and prevent watchdog to run correctly. – Thanatheos Mar 25 '19 at 10:28
  • @Thanatheos And looking at the journal doesn't work? Does my answer help you a bit? If so it would be nice if you could accept it with a tick at tick on the left side. – Ingo Mar 25 '19 at 11:06
  • watchdog seem to be great idea cause /dev is in ram and allow me to read it and reboot earlier. – Thanatheos Mar 25 '19 at 12:15
  • In fact he try to reboot normaly so WD works but I got "file /tftpboot/my_ip/dc0b6fd2/start.elf not found" on my server but he must ask at "file /tftpboot/my_ip/start.elf not found" – Thanatheos Mar 25 '19 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.