You could use "Munin" with Raspbian to monitor your RPis. You need only one "Munin Master" and on each device you want to monitor, the "Munin Node" software. "Munin" is very flexible and comes with numerous plugins. But yet the setup process with Raspbian is fast and simple if you stick to the defaults.
For the complete architecture please take a look at this diagram.
A comprehensive documentation is available: Welcome to the Munin Guide.
The Master will pull the data from the Nodes and create the graphics, make them accessible via web browser (in companion with a web server) and can alert you if a defined value got hit (i.e. "if cpu temperature > 80°C then email to hostmaster").
sudo apt-get install munin
Connect Master and Node
On the Master you need to add a so called "Host Tree" for each Node:
sudo nano /etc/munin/munin.conf
address 192.168.0.1 # IPv4 of Node 1
address 192.168.0.2 # IPv4 of Node 2
Example graphs (Munin offers zooming in 3 clicks, can't show it here):
If you have installed Munin through the distribution packages (my recommendation), the webserver will be configured automatically and available at
http://ip/munin (replace "ip").
You can configure it like any other virtual host, i.e. you can make it accessible per SSL/TLS only, protect it with credentials etc., see Authentication and Authorization - Apache HTTP Server Version 2.4.
Munin works great with Apache. lighttpd and nginx are supported as well. Webserver Configuration - Munin Guide
If you have an MTA already up and running (like Exim), you can tell Munin to alert you via email:
sudo nano /etc/munin/munin.conf
If wanted you can define for each Node a different contact in the above mentioned Host Tree:
To "Let Munin croak alarm" many other ways are possible (like syslog alert, external scripts). For details please read the comprehensive documentation.
Each device you want to monitor is a Node.
sudo apt-get install munin-node
Give each Node a unique name and grant the Master access to this Node:
sudo nano /etc/munin/munin-node.conf
host_name YourNodeNo1NameHere # name of this Node
allow ^192\.168\.0\.0$ # IPv4 of Master in RegEx format
After editing restart the node:
sudo systemctl restart munin-node.service
On a Node you can run
sudo munin-node-configure --suggest to get a list of the available, used and unused plugins (please be patient, on a Raspberry Pi 3 this command takes a few minutes to finish).
Plugins can vary from simple to complex, please consult the plugin documentation. If you want to edit a plugin, I'd recommend to read "How to write Munin plugins".
To debug a plugin you can run
sudo munin-run --debug [pluginName] (replace "[pluginName]").
To deactivate a plugin run
sudo unlink /etc/munin/plugins/[pluginName] (replace "[pluginName]"). This will not delete the plugin file, just the symbolic link. Then run
sudo systemctl restart munin-node.service and Munin will no longer use this plugin.
Troubleshooting and Logfiles
Make sure that no firewall is blocking the communication. To debug check the logfiles on the respective devices:
sudo tail -f /var/log/munin/munin-update.log
sudo tail -f /var/log/munin/munin-node.log
An official "Troubleshooting Tutorial" is available.
Raspberry Pi specific
By default the Master collects the data every 5 minutes. If your Master is under heavy workload, adjust the interval.
I recommend to use a fast SD card on the Master. For each Node this process takes around 10-15 seconds (on our setup, wifi only).