How to keep watchdog timer running during reboot/shutdown

Question

In my research so far I've noticed there are two approaches to activating a watchdog timer on raspberry: using systemd or installing watchdog. It's all explained well here: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=147501#

And I've managed to get it working both ways, but not during reboot/shutdown. I used a forkbomb to test it and it does react to that, however when i try:

sudo poweroff

It just stays down. So I'm thinking is there way to force watchdog to keep running during reboot/shutdown so it resets it after a few seconds if it doesn't boot properly?

Most approaches start watchdog during boot, but what if you need it running all the time? My raspberry once stopped in reboot and I had to manually unplug/plug it to get it working.

---

UPDATE:

Some stuff is much clearer to me now. There are 3 things to consider:

1. watchdog timer
2. watchdog daemon
3. watchdog service daemon

watchdog timer: a register that triggers a reset when it overflows, each cycle of RPI it increments, needs to be cleared (kicked) regularly to avoid reset

To run it on boot add following to /boot/config.txt:

dtparam=watchdog=on

watchdog daemon: a process responsible for clearing (kicking) the watchdog timer regularly

Install it:

sudo modprobe bcm2835_wdt echo "bcm2835_wdt" | sudo tee -a /etc/modules sudo apt-get install watchdog sudo update-rc.d watchdog defaults

Configure /etc/watchdog.conf:

watchdog-device = /dev/watchdog watchdog-timeout = 14 realtime = yes priority = 1 max-load-1 = 24

Configure /etc/default/watchdog:

watchdog_module="bcm2835_wdt"

To configure it add following to /etc/modprobe.d/watchdog.conf:

options bcm2835_wdt nowayout=1 heartbeat=10

refernce: http://vk5tu.livejournal.com/35721.html

The heartbeat parameter to the kernel module is the maximum gap between heartbeats seen by the device before the hardware reboots

The nowayout parameter determines what happens when the /dev/watchdog device is closed: is a heartbeat still expected or not? A value of 1 says that the countdown to a reboot keeps running and if the device is not reopened and a heartbeat written then the machine will reboot.

Enable it (If this doesn't work, first take care of watchdog.service and try again):

sudo systemctl enable watchdog

Monitor it:

sudo systemctl status watchdog

watchdog service daemon: acitvates when watchdog daemon fails/stops, usually runs a keepalive daemon which kicks the watchdog regularly

Configure /lib/systemd/system/watchdog.service:

# OnFailure=wd_keepalive.service - Commenting this out disables the keepalive daemon so once watchdog daemon fails/stops nothing will stop watchdog timer from rebooting RPI

[Install] WantedBy=multi-user.target - bug fix

---

Finally:

Reboot RPI. Once raspberry is rebooted, everything should start (watchdog timer and watchdog daemon). To check this use:

cat /var/log/syslog | grep watchdog

Expected output:

Jun 14 12:09:08 raspberrypi systemd[1]: Starting watchdog daemon... Jun 14 12:09:08 raspberrypi watchdog[813]: starting daemon (5.14): Jun 14 12:09:08 raspberrypi watchdog[813]: int=1s realtime=yes sync=no soft=no mla=24 mem=0 Jun 14 12:09:08 raspberrypi watchdog[813]: ping: no machine to check Jun 14 12:09:08 raspberrypi watchdog[813]: file: no file to check Jun 14 12:09:08 raspberrypi watchdog[813]: pidfile: no server process to check Jun 14 12:09:08 raspberrypi watchdog[813]: interface: no interface to check Jun 14 12:09:08 raspberrypi watchdog[813]: temperature: no sensors to check Jun 14 12:09:08 raspberrypi watchdog[813]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no Jun 14 12:09:08 raspberrypi watchdog[813]: watchdog now set to 14 seconds Jun 14 12:09:08 raspberrypi watchdog[813]: hardware watchdog identity: Broadcom BCM2835 Watchdog timer Jun 14 12:09:08 raspberrypi systemd[1]: Started watchdog daemon.

And:

ps aux|grep watchdog

Expected output:

root 813 0.0 0.2 1888 1760 ? SLs 12:09 0:00 /usr/sbin/watchdog pi 900 0.0 0.2 4752 1992 pts/0 S+ 12:10 0:00 grep --color=auto watchdog

---

TEST :

Kill watchdog daemon, run:

ps aux|grep watchdog

See process ID and kill it:

root 812 0.0 0.2 1888 1760 ? SLs 12:16 0:00 /usr/sbin/watchdog pi 898 0.0 0.2 4752 1992 pts/0 S+ 12:16 0:00 grep --color=auto watchdog

sudo kill -9 812

Classic forkbomb test, run:

: (){ :|:& };:

To test will raspberry reset if reboot process fails, run:

sudo poweroff - with this one I've had some problems

Raspberry should reboot after cca 15s.

Answer 1

sudo poweroff means shut down and stay off. The watchdog does not, and should not, intercept this. The behaviour you are experiencing is correct and you should not expect the watchdog to restart a powered-off system.

I don't think this is what you are wanting to do though. If you want to restart the pi. you should use the command sudo reboot instead. And in this case, if something hangs during the shutdown - startup sequence which reboot triggers, the watchdog should then take over.

Answer 2

I didn't wade through all the rant you linked, but the author doesn't seem to understand the watchdog.

There is a BCM hardware watchdog; If you want to start the hardware watchdog include dtparam=watchdog=on in /boot/config.txt

In and of itself this does little, although it should restart the system if not "kicked" regularly. You can write code which opens /dev/watchdog to kick it off.

There is also a watchdog daemon which you can configure to activate the watchdog; you should be able to start with sudo systemctl enable watchdog

NOTE the bug mentioned below was in Jessie and is fixed in Stretch
Unfortunately there is known bug in the systemd script, which is easily fixed. See http://unix.stackexchange.com/questions/346224/problem-with-systemd-starting-watchdog

Answer 3

Power is a system service/daemon in pi and is string parsed by an IC directly pin out/fanouted to the Broadcom IC. Operation workflow can be altered by referring to system files but from what I can tell you are trying to invoke periperhials in custom power regulation scenarios within the concept of interrupts and watchdog timers. You can alter some portion of this behavior by repinning the regulating IC and doing some pre-boot bare metal programming. Check out these and don’t hesitate to contact me if you have something in mind. check out ARM Corp. Assembly reference line instruction set as well as architecture documentation. It’s on the web.

http://infocenter.arm.com/help/topic/com.arm.doc.dui0489f/DUI0489F_arm_assembler_reference.pdf

https://www.arm.com/files/pdf/CortexM3_programming_for_ARM7_developers.pdf

Btw, say “hi” to your professor...

Answer 4

Just to help anyone here from Google, the most current way (as of 2020) of using the watchdog timer is to configure it through systemd. This doesn't require any additional software to be installed.

To do this, edit /etc/systemd/system.conf and set RuntimeWatchdogSec to your preferred value, like 30s for 30 seconds. Run systemctl daemon-reload to apply the change.

You can use wdctl to confirm the watchdog is available, which gives you a little more info than looking to see if /dev/watchdog is present.

You only need to set dtparam=watchdog=on in /boot/config.txt if you are using a Pi 2 or earlier as it's on by default since the Pi 3.