We're trying to build a system where a master Pi 3B+ controls a dozen (or more) Pi Zero W clients - all running the latest raspbian (as of 18:00 EDT today) - in an isolated WiFi island (no connectivity to the outside world). The 3B+ is set up as a wireless router/DNS/DHCP/NTP server (using hostapd and dnsmasq, and a real-time clock chip as the local time standard). Clients (Pi0W, laptops, etc.) can connect communicate successfully to this private wireless lan, and DNS/DHCP/NTP are all working as expected. However, I see consistent packet loss for traffic from the 0W to the 3B+ -- but no loss in the reverse direction, nor do other clients have problems in either direction. Testing is done with the systems minimally loaded and only the 0W and 3B+ on the wireless network.
Packet captures of extended series of pings show periods of 1-3 minutes where there's no loss, follow by periods of 1-2 minutes where only every third packet gets through (i.e. the sequence is "pass, drop, drop, pass, drop, drop, pass, ...") with occasional bursts of four or six drops in a row. I do see more ARP traffic than expected - tests with a laptop client show ARPs every 40 seconds, while with just the Pis running they're at irregular and more frequent intervals. It's also clear that NTP and ARP packets are dropped the same as the ICMP packets.
Looking at hostapd logs shows no apparent correlation between the above and wifi events (re-key, etc.); there are hints that the start of a lossy period correlates with NTP requests from the 0W, but this isn't a consistent trigger.
Per some suggestions I've found, I've disabled power management on the wlan0 interfaces at both ends, but this doesn't appear to have made any difference.
I was a network guru for 20 years before I retired, but back then it was AppleTalk and FDDI and ISDN and T1 (at first) up through 100G ethernet and MPLS - but all over copper or fiber, I didn't have much exposure to wireless (other helping the WiFi guys keep 6000+ access points on line); so I'm running out of ideas.
Can anyone either: 1) Replicate this environment and problem, or 2) Replicate the setup and NOT reproduce the problem, or 3) Point me to other straws to grasp?
FWIW I've also posted this to raspberry.org:
https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=243092
Solutions in either forum will be copied (by me) to the other.
============
After more testing I've isolated the cause: NTP queries outbound from the client trigger the network to switch between passing all traffic and passing only every third packet! In extended tests, the behavior switch always happened within a few seconds of the Zero W sending an NTP packet (although occasional NTP packet didn't trigger a switch).
Why this happens is a mystery to me - I suspect a bug deep in the IP stack or wireless driver - so I can't say this is solved yet; but I have a pretty good idea of what to try now to work around this.
==========
Since it's outbound NTP from the Pi0W that triggers the problem on it, I had hoped that I could work around this by putting the master into NTP broadcast mode and making the 0W an broadcast client. But it turns out that the broadcast client still generates some unicast traffic to the server (so the client can calculate the round-trip time); and if I drop this outbound traffic with an iptables rule, then ntpd ignores the broadcast updates.
Every other method I've found - ntpdate, timesyncd, system-timedatectl, chrony - all use NTP as the underlying transport, so these won't work either; and rdate (anyone else old enough to remember this?) has been deprecated for decades.
At this point I'll probably write a little python code to run on the zeros that will listen for "set system clock to XXX" broadcast messages from the master, and add some front-end code on the master to broadcast this every time we fire up the control program for the system.
And does anyone know how to get this to the appropriate developers so they can look into the root cause of this? I've got lots of logs, packet captures, etc. that I can share, but I don't know who to send 'em to...
