RPi irresponsive ssh access to LAN

Question

I have the following setup at home:

• 3 RPis connected to router via ethernet. They have more than enough power supplies and are headless.
  • RPi #1 (very old model) runs pihole and acts as DHCP server.
  • RPi #2 (very old model) runs some lightweight web services.
  • RPi #3 (model 4) runs some heavier web services.
• IP addresses are static as configured in RPi #1's pihole DHCP server. The router's DHCP server is switched off accordingly.

The issue I've been having for a couple of months is the following: the third RPi becomes irresponsive often and seems to freeze SSH access to any of the other RPis. Freeze as in I can ping any of the RPis fine but cannot ssh into any of them—terminal gets stuck without showing any output. RPi #3's web services become inaccessible, but the other two RPi's web services are working fine and are accessible, yet I cannot ssh into them. I pull the plug on RPi #3 and I can then ssh into the other two RPis fine. Interestingly, even when everything is frozen, I can use WireGuard on RPi #3 fine, so this RPi hasn't crashed and it has a network connection, even though all web services are seemingly down.

I use Mac OS over wifi to connect to the RPis.

I also activated the watchdog for the problematic RPi #3 but no luck.

Any ideas?

Answer 1

The router's DHCP server is switched off accordingly

You should explicitly add the static assignments to the router's configuration if possible. Otherwise, there is the possibility of the addresses to half stop working -- a node can pass traffic back and forth through the router, but the router does not pass traffic from wifi nodes to the ethernet nodes because it has no routing table entry for them¹ -- I'm guessing by "freeze" you mean attempts to reach them just time out as if nothing is there (vs. the request immediately being refused).

What you describe fits that scenario; Pi #3 is still working in the sense that wireguard tunnels are fine (because it can still pass traffic back and forth to the internet through the router, presuming that's where those tunnels go), and it can still communicate with the other Pi's because they are all on the ethernet segment, which functions like a switch. But communication between wifi nodes and ethernet nodes must be forwarded by the router.

You could confirm this by running wireshark promiscuously on one of the ethernet nodes; packets from wifi addresses will simply stop appearing.²

seems to freeze SSH access to any of the other RPis ... I pull the plug on RPi #3 and I can then ssh into the other two RPis fine

That it seems to be so is fine, but this is not really possible (ie., #3 cannot cause such a freeze) unless #3 has launched some kind of DOS attack on the other two. That would show up very clearly in wireshark, but it is pretty unlikely -- even if there were malware or something in play, I don't think one Pi could effect this on two others simultaneously.

---

1. "But," you say, "it was working 10 minutes ago!" Keep in mind the routing table is periodically updated on the basis of ARP requests. Why that should go wrong I'm not sure, but I have seen this happen. It could be, eg., that the router simply does not make such requests about addresses that are not in its list of active leases or static assignments. Hence the need for such.

2. That pings go through I don't have an explanation for, but ICMP is a distinct protocol from ARP or TCP and may get effectively broadcast across all the router's interfaces.