4

I have PIs that use a schedule to play videos on loop.

The PIs also use a daemon file 'surround.sh' to control the video player, and the daemon script allows the user to check the service status of each surround.

How can I set up a dashboard (preferably a web page) that monitors the processes running on the PI and alerts me of any common issues with the PI. I.e. if it has stopped playing a video, if the PI lost connection etc.

  • I've been looking at the munin monitoring tool and that seems to only work with one PI? I need a monitoring tool to monitor multiple raspberry PIs that are running the same daemon script (service). Is there any way I can achieve this? – Kai Mar 1 '18 at 12:50
  • Ah right! Is there a tutorial or any other resource that you're aware of which might help me integrate Munin Master with my PIs? – Kai Mar 1 '18 at 15:22
  • Thanks for the detailed answer!! I'll give your solution a go and see how I get on :) – Kai Mar 2 '18 at 9:14
5

You could use "Munin" with Raspbian to monitor your RPis. You need only one "Munin Master" and on each device you want to monitor, the "Munin Node" software. "Munin" is very flexible and comes with numerous plugins. But yet the setup process with Raspbian is fast and simple if you stick to the defaults.

For the complete architecture please take a look at this diagram.

A comprehensive documentation is available: Welcome to the Munin Guide.

The Master

The Master will pull the data from the Nodes and create the graphics, make them accessible via web browser (in companion with a web server) and can alert you if a defined value got hit (i.e. "if cpu temperature > 80°C then email to hostmaster").

To install: sudo apt-get install munin

Connect Master and Node

On the Master you need to add a so called "Host Tree" for each Node:

sudo nano /etc/munin/munin.conf

[YourNodeNo1NameHere]
    address 192.168.0.1 # IPv4 of Node 1
    use_node_name yes

[YourNodeNo2NameHere]
    address 192.168.0.2 # IPv4 of Node 2
    use_node_name yes

Dashboard

Example graphs (Munin offers zooming in 3 clicks, can't show it here):

Processes - by day

Memory usage - by month

If you have installed Munin through the distribution packages (my recommendation), the webserver will be configured automatically and available at http://ip/munin (replace "ip").

You can configure it like any other virtual host, i.e. you can make it accessible per SSL/TLS only, protect it with credentials etc., see Authentication and Authorization - Apache HTTP Server Version 2.4.

Munin works great with Apache. lighttpd and nginx are supported as well. Webserver Configuration - Munin Guide

Alerts

If you have an MTA already up and running (like Exim), you can tell Munin to alert you via email:

sudo nano /etc/munin/munin.conf

[YourNodeNo1NameHere]
    contacts email@example.invalid

If wanted you can define for each Node a different contact in the above mentioned Host Tree:

[YourNodeNo2NameHere]
    contacts other@example.invalid

To "Let Munin croak alarm" many other ways are possible (like syslog alert, external scripts). For details please read the comprehensive documentation.

The Nodes

Each device you want to monitor is a Node.

To install: sudo apt-get install munin-node

Give each Node a unique name and grant the Master access to this Node:

sudo nano /etc/munin/munin-node.conf

host_name YourNodeNo1NameHere # name of this Node
allow ^192\.168\.0\.0$ # IPv4 of Master in RegEx format

After editing restart the node: sudo systemctl restart munin-node.service

Plugins

On a Node you can run sudo munin-node-configure --suggest to get a list of the available, used and unused plugins (please be patient, on a Raspberry Pi 3 this command takes a few minutes to finish).

Plugins can vary from simple to complex, please consult the plugin documentation. If you want to edit a plugin, I'd recommend to read "How to write Munin plugins".

To debug a plugin you can run sudo munin-run --debug [pluginName] (replace "[pluginName]").

To deactivate a plugin run sudo unlink /etc/munin/plugins/[pluginName] (replace "[pluginName]"). This will not delete the plugin file, just the symbolic link. Then run sudo systemctl restart munin-node.service and Munin will no longer use this plugin.

Troubleshooting and Logfiles

Make sure that no firewall is blocking the communication. To debug check the logfiles on the respective devices:

sudo tail -f /var/log/munin/munin-update.log

sudo tail -f /var/log/munin/munin-node.log

An official "Troubleshooting Tutorial" is available.

Raspberry Pi specific

By default the Master collects the data every 5 minutes. If your Master is under heavy workload, adjust the interval.

I recommend to use a fast SD card on the Master. For each Node this process takes around 10-15 seconds (on our setup, wifi only).

3

In addition to turn-key applications, there are other simplistic ways to get diagnostic information through the web by leveraging the operating system.

SOCAT port monitor

socat can be used to listen on a port, and execute/echo a command to that port whenever a connection is made. For example, in two lines

socat tcp4-listen:5000,reuseaddr,fork system:'uptime'

on another computer

$ nc my-pi.ip 5000
09:08:21 up 6 days,  1:35,  9 users,  load average: 0.07, 0.28, 0.25

Magic!

Replace uptime with other system utilities or shell scripts that print information you need. The connection is bi-directional so you can use it to pass arguments or diagnostic commands to your scripts as well.

Use access control to restrict who can connect to this port, if this is a pi on a public network.

Aggregating information from several pis can be done with an intermediate webserver that queries each pi through a secure network or VPN and displays the needed information. Because it's a simple socket interface, you just connect and read a status string.

CGI Script

By installing nginx, apache, or other webserver, you can execute simple shell scripts to create plain text pages, this is usually sufficient for internal (non-public) diagnostic and debug information.

It is not recommended to use this for devices on the public internet, however, security risks can be mitigated by using access control lists in the webserver (ACL) to limit hosts that can connect to the pi.

Script Example (/var/www/cgi-bin/proccheck, make sure it is executable)

#!/bin/bash
echo "Content-type: text/html"
echo ""
echo '<html>'
echo '<head>'
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">'

PATH="/bin:/usr/bin:/usr/opt/bin"
export $PATH

echo '<title>System Monitor</title>'

echo '</head>'
echo '<body>'
echo '<h3>'

hostname

echo '</h3>'


# Whatever monitoring shell commands, e.g. cat a log file, check a /proc/ node, print system load/etc
ps -A


echo '</body>'
echo '</html>'

exit 0

go to http://my-pi/cgi-bin/proccheck

CSS can be added through additional echo statements.

  • There are other fun, low effort tricks. 'netcat' can be used to create a broadcast peer to peer chat room between clients, simply have a single chat room where each pi is spamming state information on a timer. You monitor the chat room or forward it to slack or email – crasic Mar 1 '18 at 17:26
  • I will give your solution a go! – Kai Mar 2 '18 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.