Very odd and frustrating behaviour of multiple raspberry pi becoming unresponsive

Question

I'm getting really frustrated with this, I hope someone can offer some guidance.

I have 2 raspberry Pis - one in my living room and one in my garage. Over the last 2 weeks or so they have both started becoming unresponsive after some time (time varies). I have tried re-imaging them and replacing sd cards.

They are both raspberry pi 3 and are both on wifi. They are both headless.

Sometimes when I power them up they never become available for connection on ssh. Sometimes they are available for a short time or a few hours then I can no longer ssh to them.

Today I brought one of them up to the study where I have a monitor that I can plug in. Of course it booted fine and all was perfect. I moved it down to the living room and plugged it and and I could never connect to it. I tried moving it around a bit thinking that the wifi signal could be the issue (it's in a tv cabinet) despite my phone reporting a very string signal. This seemed to help and I got it connected (I now think that the wifi signal was a red herring).

After I successfully connected to it I sat with a terminal open all day pinging both pi and they both stayed alive all day.

I went out to the Gym for an hour and come home and they are now both unresponsive again. I tried rebooting them in the same place that they have been responding to pings all day and I am still not able to connect to them.

When they are not responding to SSH I can't see them on the attached devices list on my router.

Does anyone have any suggestions about how to start investigating this?

Answer 1

Does anyone have any suggestions about how to start investigating this?

You need to verify from the Pi whether it is remaining connected or not via some kind of simple script, e.g.

#!/bin/bash

exec &>> /var/log/ping-test.log

pingIP=192.168.0.1
# Seconds, i.e., 5 minutes
pingInterval=300

echo $(date +"%D %T")" Started, PID $BASHPID"

while [ 1 ]; do
        echo -n $(date +"%D %T")" "
        ping -c 1 $pingIP &> /dev/null
        if [ $? -ne 0 ]; then
                echo "Ping failed."
        else
                echo "Ping OK."
        fi
        sleep $pingInterval
done

The examplepingIP I've used is a common one for LAN routers, which is a good initial choice because it does prove whether the wifi is up (you can test first manually to make sure your router will ping back, almost certainly it will).

However, I have noticed that the DHCP lease can expire and if for some reason it isn't renewed properly (can happen if, e.g., the node starts up after being offline for a week, etc.), pinging the router may still succeed even on WPA encrypted WLANs (since WPA authentication is a separate issue), but the router won't route anything out and even reaching other nodes on the local network becomes a crap shoot.

So if after you notice the unit go offline you find it was still logging "Ping OK" (maybe you want a longer interval than 5 minutes), then you know the wifi is up but something else weird is going on.

The problem I've had along these lines, already hinted at, is DHCP leases that are mysteriously allowed to expire and not get renewed. In any case, at that point you want to try something outside the LAN, and use an actual numerical IP, not a domain name, since that depends on access to your DNS servers which just adds confusion (you can test both if this is an issue). One potential target is google's public DNS servers, 8.8.8.8 (and 8.8.4.4), which respond to pings, or your actual ISP nameserver -- but as far as I am aware you need to extract that from /etc/resolv.conf (which is not hard but let's stop here).

If that fails but pinging the router works, your Pi is online but the router is not routing it.

To run this, log in and:

sudo setsid ./ping-test.sh &

Note the PID is logged when the script starts, which makes it easy to check for and kill. You can also start this from /etc/rc.local:

/path/to/ping-test.sh &

The ampersands here are critical. Also, if you are not used to using shell scripts, they need to be executable when invoked via a shebang (the first #! line) -- chmod 755 ping-test.sh.

There's obviously all kinds of other information you can include here if you want. The exec &>> line at the top means all output will be redirected to /var/log/ping-test.log. An example of the output from this:

08/12/16 15:43:03 Started, PID 31515
08/12/16 15:43:03 Ping OK.