1

I have a headless Raspberry Pi sitting in a corner, and I'd like to know if something's gone wrong with it, like it's lost a network connection or crashed. I don't need a complex monitoring or logging system, I just want a basic heart beat mechanism to get an email or other notification when something goes wrong.

1
  • I see a couple close votes for "Needs more focus"; would anyone care to clarify what's lacking about the question? I'm happy to expand on it if something's unclear.
    – dimo414
    Mar 18 '20 at 1:08
2

https://healthchecks.io/ is a great tool for basic monitoring and alerting, and works perfectly for a heartbeat check. Simply create a check with a relatively frequent period (I went with a 30 minute period and 15 minute grace, meaning a single skipped beat won't alert, but two successive will), and then configure a cron job to ping the check every 20 minutes:

#      m h dom mon dow command
10,30,50 * *   *   *   curl -fsS --retry 3 https://hc-ping.com/YOUR_HEALTH_CHECK_ID > /dev/null

This will alert if either the network or machine go down for more than an hour.


healthchecks.io also has a neat feature that supports including a small (10kb) payload with the heartbeat, which seems like a nice way to include some basic status details about the machine as well. I created a small script that sends a payload of basic system diagnostics, and will also fail the check on a few other conditions such as not enough disk space.

I'm still experimenting with exactly what information and checks are worth incorporating, I'd welcome any feedback: https://gist.github.com/dimo414/10d6f162fb6d72f517a041d28d92314f

1

On Debian/Raspbian the default application to monitor a system is watchdog. It is available from the repository and can simply installed with:

rpi ~$ sudo apt install watchdog

By default it will reboot the system if some configurable tests fail. But on the man page of watchdog you will find that it is also possible to send an Email:

If the machine is halted an email is sent to notify a human that the machine is going down. Starting with version 4.4 watchdog will also notify the human in charge if the machine is rebooted.

5
  • 1
    Probably worth noting that "sending an email" this way is very far from trivial, unless it is to a user on the same machine. Beyond that you need the services of a bona fide email server. Which does not exist on the Pi by default, so you would need to make arrangements...nothing in the man page about that (man watchdog.conf is no more helpful), which implies it uses the system sendmail, which, again, making that work is going to be a lot of work.
    – goldilocks
    Mar 17 '20 at 17:17
  • @goldilocks Then the problem with sending an email always persist, no matter what application do you use. It seems to be a general problem.
    – Ingo
    Mar 17 '20 at 19:51
  • The problem with any sort of local monitoring utility is it's dependent of the system being in a healthy enough state to execute the watchdog in the first place. For example if the system loses its network connection no email notifications will be getting out :)
    – dimo414
    Mar 18 '20 at 1:07
  • @dimo414 watchdog is supported by the kernel. It's not necessary that watchdog runs as long as the kernel itself runs. If watchdog fails it does not trigger the kernel and the kernel do what it can in the failing state, mostly just reboot.
    – Ingo
    Mar 18 '20 at 10:18
  • Sure, but there are plenty of situations (again, such as a bad network config) that are not fixed by a reboot. I appreciate the pointer to watchdog, but it just doesn't seem appropriate for the goal of notifying when the device is in a bad state. watchdog can attempt to fix the problem, but if it's unable to do so you're no better off than if you hadn't set it up at all.
    – dimo414
    Mar 18 '20 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.