I'm working on a project that makes use of Raspberry Pis which will be deployed to several locations. They have special hardware attached to it for automation purposes, and run a web-service (that I wrote) to access the hardware. We will occasionally deploy new software to the Pis - method yet to be determined - and this is a big area of concern.
My question: Can you name some ideas or best practices to mitigate need for manual intervention in case of issues? I'm wondering if there's any type of watchdog solution whereby the Pi does a normal reboot in case of a hang (maybe inability to phone home?), and if it still is non-functional, reboots into a fail-safe partition that can "phone home" for a solution (either reimage, or do a normal reboot again if there was just a network outage).
I'm really not sure exactly what I'm looking for, just ways that will prevent us from needing to call or send people to do manual reboots or reimages. A dual partitioning scheme seemed like it may be helpful, but maybe there are better ideas.