Raspberry Pi Stack Exchange is a question and answer site for users and developers of hardware and software for Raspberry Pi. It only takes a minute to sign up.
Sign up to join this community
I have PIs that use a schedule to play videos on loop.
The PIs also use a daemon file 'surround.sh' to control the video player, and the daemon script allows the user to check the service status of each surround.
How can I set up a dashboard (preferably a web page) that monitors the processes running on the PI and alerts me of any common issues with the PI. I.e. if it has stopped playing a video, if the PI lost connection etc.
You could use "Munin" with Raspbian to monitor your RPis. You need only one "Munin Master" and on each device you want to monitor, the "Munin Node" software. "Munin" is very flexible and comes with numerous plugins. But yet the setup process with Raspbian is fast and simple if you stick to the defaults.
For the complete architecture please take a look at this diagram.
A comprehensive documentation is available: Welcome to the Munin Guide.
The Master will pull the data from the Nodes and create the graphics, make them accessible via web browser (in companion with a web server) and can alert you if a defined value got hit (i.e. "if cpu temperature > 80°C then email to hostmaster").
To install: sudo apt-get install munin
On the Master you need to add a so called "Host Tree" for each Node:
sudo nano /etc/munin/munin.conf
[YourNodeNo1NameHere]
address 192.168.0.1 # IPv4 of Node 1
use_node_name yes
[YourNodeNo2NameHere]
address 192.168.0.2 # IPv4 of Node 2
use_node_name yes
Example graphs (Munin offers zooming in 3 clicks, can't show it here):
If you have installed Munin through the distribution packages (my recommendation), the webserver will be configured automatically and available at http://ip/munin (replace "ip").
You can configure it like any other virtual host, i.e. you can make it accessible per SSL/TLS only, protect it with credentials etc., see Authentication and Authorization - Apache HTTP Server Version 2.4.
Munin works great with Apache. lighttpd and nginx are supported as well. Webserver Configuration - Munin Guide
If you have an MTA already up and running (like Exim), you can tell Munin to alert you via email:
sudo nano /etc/munin/munin.conf
[YourNodeNo1NameHere]
contacts [email protected]
If wanted you can define for each Node a different contact in the above mentioned Host Tree:
[YourNodeNo2NameHere]
contacts [email protected]
To "Let Munin croak alarm" many other ways are possible (like syslog alert, external scripts). For details please read the comprehensive documentation.
Each device you want to monitor is a Node.
To install: sudo apt-get install munin-node
Give each Node a unique name and grant the Master access to this Node:
sudo nano /etc/munin/munin-node.conf
host_name YourNodeNo1NameHere # name of this Node
allow ^192\.168\.0\.0$ # IPv4 of Master in RegEx format
After editing restart the node: sudo systemctl restart munin-node.service
On a Node you can run sudo munin-node-configure --suggest to get a list of the available, used and unused plugins (please be patient, on a Raspberry Pi 3 this command takes a few minutes to finish).
Plugins can vary from simple to complex, please consult the plugin documentation. If you want to edit a plugin, I'd recommend to read "How to write Munin plugins".
To debug a plugin you can run sudo munin-run --debug [pluginName] (replace "[pluginName]").
To deactivate a plugin run sudo unlink /etc/munin/plugins/[pluginName] (replace "[pluginName]"). This will not delete the plugin file, just the symbolic link. Then run sudo systemctl restart munin-node.service and Munin will no longer use this plugin.
Make sure that no firewall is blocking the communication. To debug check the logfiles on the respective devices:
sudo tail -f /var/log/munin/munin-update.log
sudo tail -f /var/log/munin/munin-node.log
An official "Troubleshooting Tutorial" is available.
By default the Master collects the data every 5 minutes. If your Master is under heavy workload, adjust the interval.
I recommend to use a fast SD card on the Master. For each Node this process takes around 10-15 seconds (on our setup, wifi only).
In addition to turn-key applications, there are other simplistic ways to get diagnostic information through the web by leveraging the operating system.
socat can be used to listen on a port, and execute/echo a command to that port whenever a connection is made. For example, in two lines
socat tcp4-listen:5000,reuseaddr,fork system:'uptime'
on another computer
$ nc my-pi.ip 5000
09:08:21 up 6 days, 1:35, 9 users, load average: 0.07, 0.28, 0.25
Magic!
Replace uptime with other system utilities or shell scripts that print information you need. The connection is bi-directional so you can use it to pass arguments or diagnostic commands to your scripts as well.
Use access control to restrict who can connect to this port, if this is a pi on a public network.
Aggregating information from several pis can be done with an intermediate webserver that queries each pi through a secure network or VPN and displays the needed information. Because it's a simple socket interface, you just connect and read a status string.
By installing nginx, apache, or other webserver, you can execute simple shell scripts to create plain text pages, this is usually sufficient for internal (non-public) diagnostic and debug information.
It is not recommended to use this for devices on the public internet, however, security risks can be mitigated by using access control lists in the webserver (ACL) to limit hosts that can connect to the pi.
Script Example (/var/www/cgi-bin/proccheck, make sure it is executable)
#!/bin/bash
echo "Content-type: text/html"
echo ""
echo '<html>'
echo '<head>'
echo '<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">'
PATH="/bin:/usr/bin:/usr/opt/bin"
export $PATH
echo '<title>System Monitor</title>'
echo '</head>'
echo '<body>'
echo '<h3>'
hostname
echo '</h3>'
# Whatever monitoring shell commands, e.g. cat a log file, check a /proc/ node, print system load/etc
ps -A
echo '</body>'
echo '</html>'
exit 0
go to http://my-pi/cgi-bin/proccheck
CSS can be added through additional echo statements.
