2

We're trying to build a system where a master Pi 3B+ controls a dozen (or more) Pi Zero W clients - all running the latest raspbian (as of 18:00 EDT today) - in an isolated WiFi island (no connectivity to the outside world). The 3B+ is set up as a wireless router/DNS/DHCP/NTP server (using hostapd and dnsmasq, and a real-time clock chip as the local time standard). Clients (Pi0W, laptops, etc.) can connect communicate successfully to this private wireless lan, and DNS/DHCP/NTP are all working as expected. However, I see consistent packet loss for traffic from the 0W to the 3B+ -- but no loss in the reverse direction, nor do other clients have problems in either direction. Testing is done with the systems minimally loaded and only the 0W and 3B+ on the wireless network.

Packet captures of extended series of pings show periods of 1-3 minutes where there's no loss, follow by periods of 1-2 minutes where only every third packet gets through (i.e. the sequence is "pass, drop, drop, pass, drop, drop, pass, ...") with occasional bursts of four or six drops in a row. I do see more ARP traffic than expected - tests with a laptop client show ARPs every 40 seconds, while with just the Pis running they're at irregular and more frequent intervals. It's also clear that NTP and ARP packets are dropped the same as the ICMP packets.

Looking at hostapd logs shows no apparent correlation between the above and wifi events (re-key, etc.); there are hints that the start of a lossy period correlates with NTP requests from the 0W, but this isn't a consistent trigger.

Per some suggestions I've found, I've disabled power management on the wlan0 interfaces at both ends, but this doesn't appear to have made any difference.

I was a network guru for 20 years before I retired, but back then it was AppleTalk and FDDI and ISDN and T1 (at first) up through 100G ethernet and MPLS - but all over copper or fiber, I didn't have much exposure to wireless (other helping the WiFi guys keep 6000+ access points on line); so I'm running out of ideas.

Can anyone either: 1) Replicate this environment and problem, or 2) Replicate the setup and NOT reproduce the problem, or 3) Point me to other straws to grasp?

FWIW I've also posted this to raspberry.org:

https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=243092

Solutions in either forum will be copied (by me) to the other.

============

After more testing I've isolated the cause: NTP queries outbound from the client trigger the network to switch between passing all traffic and passing only every third packet! In extended tests, the behavior switch always happened within a few seconds of the Zero W sending an NTP packet (although occasional NTP packet didn't trigger a switch).

Why this happens is a mystery to me - I suspect a bug deep in the IP stack or wireless driver - so I can't say this is solved yet; but I have a pretty good idea of what to try now to work around this.

==========

Since it's outbound NTP from the Pi0W that triggers the problem on it, I had hoped that I could work around this by putting the master into NTP broadcast mode and making the 0W an broadcast client. But it turns out that the broadcast client still generates some unicast traffic to the server (so the client can calculate the round-trip time); and if I drop this outbound traffic with an iptables rule, then ntpd ignores the broadcast updates.

Every other method I've found - ntpdate, timesyncd, system-timedatectl, chrony - all use NTP as the underlying transport, so these won't work either; and rdate (anyone else old enough to remember this?) has been deprecated for decades.

At this point I'll probably write a little python code to run on the zeros that will listen for "set system clock to XXX" broadcast messages from the master, and add some front-end code on the master to broadcast this every time we fire up the control program for the system.

And does anyone know how to get this to the appropriate developers so they can look into the root cause of this? I've got lots of logs, packet captures, etc. that I can share, but I don't know who to send 'em to...

  • Hi @Doctor Kurt, how nice to meet a retired network guru. I am a hobbyist with 5 hobbyist years of Arduino and another 5 years of Rpi. I have 7 Rpi3B+s and 3 RpiZ/Ws, all running the latest stretch 9, and regularly upgraded and updated. You suggest (1) replicate problematic setup, (2) replicate similar, but no problem setup. I once played with RpiZ/W and my quick and dirty conclusion is that (a) it is about 3 times slower to develop app, cf Rpi. (b) software (drivers etc) not that updated and (at least community) supported. Therefore I think suggestion (1) is a waste of time, ... – tlfong01 Jun 21 at 1:46
  • / cont. from above. However your suggestion (2) is good. Similar setups can be: (a) ALL Rpi's (forget ALL slow and stupid RpiZw's for now), with built in WiFi or external dongles with "long" antennas. (b) Fall back to older, but should be more reliable Raspbians (stretech 9 2018apr08 is too new and drivers might not that compatible, ... (c) If all Rpi's setup OK, then add in stupid Z (with strong W dongles) or ZWs. I am only suggesting things on system/hardware side. But of course the packet loss is software/config related. PS. all above is only a hobbyist's opinion, ... :) – tlfong01 Jun 21 at 1:58
  • FWIW we're using the Pi0W because we need something that's compact, light weight and affordable - we don't need high performance. We're starting with 12 of them, but the full system might increase the count to 30 (or more). Us retirees have to be careful with our bank accounts... – Doctor Kurt Jun 21 at 2:16
  • I fully agree with you selecting cheapy PioW. I suggested all Rpi is ONLY for trouble shooting and R&D. Of course using Rpi0W is not just to save money, but save space and other resources (using appropriate technology, and thus environmental friendly, ie) . One other reason I don't like Pi0W because it is "expensive", cf esp8266: Immediately I found Rpi0W so stupid, I switched to ESP8266 Wifi Controller. Sadly, ESP8266 is not that mature, and NodeMCU Lua is weak. I am waiting for Rpi4 and ESP32, STM32, to mature. Looking back, my first user experience with Rpi0W is indeed painful! :) – tlfong01 Jun 21 at 2:36
  • Y comnt on NTP reminds me: When first tried, pi0w, I found it not recognize my expensive and reliable USB hub which I have been using happily for a couple of years for Rpi 1/2/3. I need to find another cheapy HUB by trials and errors. And as Warren Buffett says: "There’s never just one cockroach in the kitchen". That is one of the main reasons I gave up pi0W. Actually I bought pi0W because Google says its AI hat only works on pi0W. I tried it and disappointedly found Google AI cannot make use of the pi0W's GPU (license reasons), on top of USB2 communication bottleneck. – tlfong01 Jun 21 at 3:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.