2

I brought up a Pi Zero W last month running the then-current Raspberry Pi OS (Raspbian Buster) 20-02-27 release. It has updated itself to linux kernel 5.4.51+. This headless system is running 5 servers (of note): an RDP server (Xrdp), a VNC server (vnc), a DNS server (dnsmasq); an NTP server (ntpd), and a fifth specialty Python 3 server that accepts and responds to HTTP requests. The OS has been configured with a static IP, and static Gateway and DNS IPs. All is well and good with that configuration and the servers work fine and do their respective jobs...

That is... up to a point. As the system runs for hours, and then for days, what happens is that various client devices on my network attempting to access these servers find themselves unable to access the static IP of my Pi Zero W system. It is as if the IP is simply unaccessible (as if blocked by a firewall). The access seems to be lost to the various client devices in a pretty much random fashion over the period of a day or two until none of them can access my system. Once the Pi Zero W's IP becomes unaccessible to a given device, it stays that way...

UNTIL... I execute any sort of command such as ping or rsh that attempts an outgoing connection from the Pi Zero W system to the external device. Then, suddenly the device can again see the Pi Zero W's IP and can access the servers...

That is... for a while; then after hours to days, the ability for that device to connect breaks again. This is very frustrating.

My current workaround is to run a background shell script (on the Pi Zero W) that does a single ping (or for those devices that don't respond to ping, a single denied ssh) to each device once a minute. That mostly solves the problem, but because one of the client devices is an iPad and its WiFi shuts down when the cover is closed, I have to wait up to a minute after opening the cover (if it's been closed for a day or two) to regain access.

I will note that I also have an 2.5 year old Pi 3 Model B running the Raspbian OS from its day and running the same server configuration and external devices on my network NEVER lose the ability to communicate with it. However, the problem is not with the Pi Zero W hardware (I tried swapping in a new Pi Zero W), but rather with this version of Raspbian (and possibly its interaction with the Pi Zero W's WiFi chip). I am not aware of any firewall running in the OS (but there could be - that's why I'm asking).

So my question is: Given all of the above, can anyone help me find out how to stop this from happening other than my band-aid shell script solution? I'd settle for just an explanation of this unusual behavior even if it can't be fixed.

3
  • There USED to be a quirk where the WiFi went into low power mode. It is supposed to be fixed ages ago - maybe that's it. Pi power issues can have these symptoms (spent ages tracking that one down on my Pi 3) - See aoakley.com/articles/2014-02-25-raspberry-pi-power-wifi.php You can still get Stretch maybe try that? – user115418 Sep 13 '20 at 21:45
  • Add iwconfig wlan0 power off to /etc/rc.local (just before the exit 0 line) to disable WiFi power management. – Dougie Sep 14 '20 at 0:29
  • Nope, it's not WiFi power management. It was on but when I turned it off, that had no effect on the problem. Besides, notice that WiFi power management could only make the Pi Zero W become inaccessible from all IPs or accessible from all IPs, while this problem is device specific (it doesn't matter what IP the device has - I can even change the IP and that does not fix things - so it must be blocking based on MAC address. And pinging one device to get it going again has no effect on other devices that are blocked; each must be pinged individually to give it access again. – Frank70 Sep 14 '20 at 20:34
2

This could be an issue with ARP not being able to resolve the MAC address of your Pi Zero W. Each host maintains an ARP cache mapping IP address to MAC address for connected LAN networks.

When a host needs to connect to some LAN-connected address (say 192.168.1.1), it will first send an ARP who-has request (tcpdump output):

# tcpdump -n -i wlan0 arp
...
18:40:17.053078 ARP, Request who-has 192.168.1.1 tell 192.168.1.112, length 28

This is answered by the host (or possibly a proxy):

18:40:17.055447 ARP, Reply 192.168.1.1 is-at 7c:8b:ca:ce:81:70, length 28

ARP cache entries time out after a while, requiring the sending host to re-issue the query. Normally all this happens under the covers, but if something goes wrong - your router cache fills up, or they are firewalled in one direction, weird problems like this can ensue.

It could be that your client devices are expiring the MAC address and their ARP queries are being lost somewhere. But when you ping a client device from your Pi Zero W, your first packet from the Pi0W to that client will update the ARP cache on the client without the client having to send its own request.

I suggest you choose one client, and tcpdump all ARP and that client on your Pi Zero W until the problem recurs, to give you an idea if this is what is going on. Example:

tcpdump -i wlan0 -w outputfile.tcpdump arp or host 192.168.1.55

Run a similar tcpdump command on another host on your LAN to see the packets from a different perspective.

When your client (for example the Ipad) wakes up, you should see an ARP request from the client on your other host (as ARP requests are broadcast) and maybe your Pi Zero W doesn't see it. If the Pi0W's receiving some other WiFi traffic at the time then it's not an issue with the hardware shutting down. Check your WiFi access point settings; it is acting as a broadcaster of the ARP requests. Make sure this isn't disabled, or something else is doing proxy ARP.

I know you said your Pi Zero W is statically configured - even so, check if DHCP is not somehow involved; check that DNS resolution to your Pi Zero W is occurring correctly, unless the clients are configured to connect by IP address.

6
  • Thank you so much for providing all this information. It is a lot to absorb, but based on it I've already found threads on the internet where others have experienced similar problems. I'm going to explore this path now that you've opened my eyes. I'm so used to IP just working that I don't even think about the link layer and what needs to happen there, but now I will! I'll report back when I've found a solution, or even a cause. – Frank70 Sep 18 '20 at 12:40
  • 1
    @Frank70 Here is another description of the problem with some background info: Android device cannot access Raspberry Pi Zero W. – Ingo Sep 19 '20 at 9:01
  • Could someone explain what entity in the Raspberry Pi Zero W is responsible for replying to an ARP request? Is it software (a service or in the kernel?) or the WiFi hardware? Using tcpdump on the Raspberry Pi Zero W shows no incoming ARP requests for 192.168.1.188 (the Raspberry Pi's IP). From what entity on my network are the APR requests broadcast? So far, things that restore communication are outgoing (from the Pi) pings, rebooting the Pi, or momentarily pulling power on the WiFi access point to which it is connected. But, like the cat that came back... the issue comes back :( – Frank70 Sep 19 '20 at 18:12
  • Thank you Ingo - Tilman's problem sounds identical to mine.The mystery is why my other Raspberry Pi (a Pi3 model B) configured virtually identically but with a 2-year old Raspbian OS, experiences no such problem. – Frank70 Sep 19 '20 at 18:17
  • The kernel generates and processes the ARP requests and responses; it would be great if you can sniff the traffic of the client and the WiFi in addition to the Pi Zero W, so you can verify that the client sends an ARP request; the Access Point (does|does not) forward the ARP request; the Pi Zero W (does|does not) see the request come in over the air. – nickandrew Sep 20 '20 at 7:08
0

I believe I have found out what was causing my problem. I still don't fully understand ARP, but thanks to all those who pointed me to ARP as the likely cause, I have done an experiment that shows how I can run my server without losing connectivity from various clients. Let me explain:

I originally deployed a single copy of this server on a Raspberry Pi 3 Model B that was primarily being used for something else. The server provides 3 services to the clients: DNS, NTP, and a custom service that provides guide data to client DVRs in my home. I'll call that Pi "191", because its static IP address is 192.168.1.191. With that server running, there was never any problem.

In order to liberate that Pi for its original purpose, I purchased a Raspberry Pi Zero W, replicated all the services on that, and switched all my clients to reference the Pi Zero W. I'll call that Pi "186", because its static IP address is 192.168.1.186.

My mistake is that, even though it was no longer being used by the clients, I left "191" up and running on my network at the same time I was testing and using "186". Now each of the two servers had an entry in its /etc/hosts file equating its own IP address with the domain name epg.channelmastertv.com (this is essential to functionality of the server in its DNS role). So "191" had a /etc/hosts that said

192.168.1.191 epg.channelmastertv.com

while "186" had a /etc/hosts that said

192.168.1.186 epg.channelmastertv.com

So each server was identifying itself as epg.channelmastertv.com when it was being used by a client as a DNS server. I had noticed when using tcpdump, that ARP requests and replies were being logged with the domain name rather than the numeric IP, and I figured that somehow the two servers having epg.channelmastertv.com equated to different IPs was causing the problem, even though that association should have only been local to each particular server, and visible externally only to the DVR clients that were using a given server for DNS. But apparently it somehow screws with ARP (this is the part I really don't understand). I don't know why it always picked "186" to lose track of other than it has a much slower processor and may not respond as quickly as "191" to ARP requests.

My solution then, in simple terms, was to turn off "191" (I didn't really do that, I actually took the entry out of its /etc/hosts file and rebooted it, but the effect is the same). Then for the past 3 days, clients no longer lost their ability to connect to "186"!

So the lesson I learned is not to have two of these servers up and running on my network at the same time.

For those of you who are curious why /etc/hosts needed to be configured that way, or how the server provides guide data to the DVRs, that information can be found in this forum: PiGS: The Channel Master DVR+ EP Guide Server Replacement

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.