I have a headless Raspberry Pi sitting in a corner, and I'd like to know if something's gone wrong with it, like it's lost a network connection or crashed. I don't need a complex monitoring or logging system, I just want a basic heart beat mechanism to get an email or other notification when something goes wrong.
1
-
I see a couple close votes for "Needs more focus"; would anyone care to clarify what's lacking about the question? I'm happy to expand on it if something's unclear.– dimo414Commented Mar 18, 2020 at 1:08
Add a comment
|
2 Answers
https://healthchecks.io/ is a great tool for basic monitoring and alerting, and works perfectly for a heartbeat check. Simply create a check with a relatively frequent period (I went with a 30 minute period and 15 minute grace, meaning a single skipped beat won't alert, but two successive will), and then configure a cron job to ping the check every 20 minutes:
# m h dom mon dow command
10,30,50 * * * * curl -fsS --retry 3 https://hc-ping.com/YOUR_HEALTH_CHECK_ID > /dev/null
This will alert if either the network or machine go down for more than an hour.
healthchecks.io also has a neat feature that supports including a small (10kb) payload with the heartbeat, which seems like a nice way to include some basic status details about the machine as well. I created a small script that sends a payload of basic system diagnostics, and will also fail the check on a few other conditions such as not enough disk space.
I'm still experimenting with exactly what information and checks are worth incorporating, I'd welcome any feedback: https://gist.github.com/dimo414/10d6f162fb6d72f517a041d28d92314f
On Debian/Raspbian the default application to monitor a system is watchdog. It is available from the repository and can simply installed with:
rpi ~$ sudo apt install watchdog
By default it will reboot the system if some configurable tests fail. But on the man page of watchdog you will find that it is also possible to send an Email:
If the machine is halted an email is sent to notify a human that the machine is going down. Starting with version 4.4 watchdog will also notify the human in charge if the machine is rebooted.
-
1Probably worth noting that "sending an email" this way is very far from trivial, unless it is to a user on the same machine. Beyond that you need the services of a bona fide email server. Which does not exist on the Pi by default, so you would need to make arrangements...nothing in the man page about that (
man watchdog.confis no more helpful), which implies it uses the systemsendmail, which, again, making that work is going to be a lot of work.– goldilocks ♦Commented Mar 17, 2020 at 17:17 -
@goldilocks Then the problem with sending an email always persist, no matter what application do you use. It seems to be a general problem.– IngoCommented Mar 17, 2020 at 19:51
-
The problem with any sort of local monitoring utility is it's dependent of the system being in a healthy enough state to execute the watchdog in the first place. For example if the system loses its network connection no email notifications will be getting out :)– dimo414Commented Mar 18, 2020 at 1:07
-
@dimo414 watchdog is supported by the kernel. It's not necessary that watchdog runs as long as the kernel itself runs. If watchdog fails it does not trigger the kernel and the kernel do what it can in the failing state, mostly just reboot.– IngoCommented Mar 18, 2020 at 10:18
-
Sure, but there are plenty of situations (again, such as a bad network config) that are not fixed by a reboot. I appreciate the pointer to
watchdog, but it just doesn't seem appropriate for the goal of notifying when the device is in a bad state.watchdogcan attempt to fix the problem, but if it's unable to do so you're no better off than if you hadn't set it up at all.– dimo414Commented Mar 18, 2020 at 23:06