How can I mitigate the need for manual intervention with remotely deployed Raspberry Pis?

Question

I'm working on a project that makes use of Raspberry Pis which will be deployed to several locations. They have special hardware attached to it for automation purposes, and run a web-service (that I wrote) to access the hardware. We will occasionally deploy new software to the Pis - method yet to be determined - and this is a big area of concern.

My question: Can you name some ideas or best practices to mitigate need for manual intervention in case of issues? I'm wondering if there's any type of watchdog solution whereby the Pi does a normal reboot in case of a hang (maybe inability to phone home?), and if it still is non-functional, reboots into a fail-safe partition that can "phone home" for a solution (either reimage, or do a normal reboot again if there was just a network outage).

I'm really not sure exactly what I'm looking for, just ways that will prevent us from needing to call or send people to do manual reboots or reimages. A dual partitioning scheme seemed like it may be helpful, but maybe there are better ideas.

Answer 1

This answer is not specific to the raspberry pi, but is general advice for remote administered servers.

1. ssh. Using secure shell is almost as good as being there.

2. serial console and out of band administration: if you screw up your network interface having a second way to login is a lifesaver. a simple way would be to have a Identical backup machine running as a hot standby with a pair of serial connections between them so that each can reload the other.

3. remote power control: many data centers can powerfail nonfunctioning servers remotely. expensive but very nice.

4. email: if you install nullmailer and logwatch you get daily status reports which can help catch failures early.

5. rpi specific note: don't store /var or /tmp on a sd card, doing so will shorten the life of the sd card.