2

I have the following setup at home:

  • 3 RPis connected to router via ethernet. They have more than enough power supplies and are headless.
    • RPi #1 (very old model) runs pihole and acts as DHCP server.
    • RPi #2 (very old model) runs some lightweight web services.
    • RPi #3 (model 4) runs some heavier web services.
  • IP addresses are static as configured in RPi #1's pihole DHCP server. The router's DHCP server is switched off accordingly.

The issue I've been having for a couple of months is the following: the third RPi becomes irresponsive often and seems to freeze SSH access to any of the other RPis. Freeze as in I can ping any of the RPis fine but cannot ssh into any of them—terminal gets stuck without showing any output. RPi #3's web services become inaccessible, but the other two RPi's web services are working fine and are accessible, yet I cannot ssh into them. I pull the plug on RPi #3 and I can then ssh into the other two RPis fine. Interestingly, even when everything is frozen, I can use WireGuard on RPi #3 fine, so this RPi hasn't crashed and it has a network connection, even though all web services are seemingly down.

I use Mac OS over wifi to connect to the RPis.

I also activated the watchdog for the problematic RPi #3 but no luck.

Any ideas?

2
  • You should edit into the question: 1) How the Pi addresses are set (static or DHCP); if static, what arrangements you have made on the router to support this. 2) What you are connecting from -- the OS, how it receives an address, and whether this is via wifi or ethernet.
    – goldilocks
    Commented Feb 27, 2023 at 15:03
  • Thank you @goldilocks, edited with more info.
    – AlvaroP
    Commented Feb 27, 2023 at 16:26

1 Answer 1

1

The router's DHCP server is switched off accordingly

You should explicitly add the static assignments to the router's configuration if possible. Otherwise, there is the possibility of the addresses to half stop working -- a node can pass traffic back and forth through the router, but the router does not pass traffic from wifi nodes to the ethernet nodes because it has no routing table entry for them1 -- I'm guessing by "freeze" you mean attempts to reach them just time out as if nothing is there (vs. the request immediately being refused).

What you describe fits that scenario; Pi #3 is still working in the sense that wireguard tunnels are fine (because it can still pass traffic back and forth to the internet through the router, presuming that's where those tunnels go), and it can still communicate with the other Pi's because they are all on the ethernet segment, which functions like a switch. But communication between wifi nodes and ethernet nodes must be forwarded by the router.

You could confirm this by running wireshark promiscuously on one of the ethernet nodes; packets from wifi addresses will simply stop appearing.2

seems to freeze SSH access to any of the other RPis ... I pull the plug on RPi #3 and I can then ssh into the other two RPis fine

That it seems to be so is fine, but this is not really possible (ie., #3 cannot cause such a freeze) unless #3 has launched some kind of DOS attack on the other two. That would show up very clearly in wireshark, but it is pretty unlikely -- even if there were malware or something in play, I don't think one Pi could effect this on two others simultaneously.


  1. "But," you say, "it was working 10 minutes ago!" Keep in mind the routing table is periodically updated on the basis of ARP requests. Why that should go wrong I'm not sure, but I have seen this happen. It could be, eg., that the router simply does not make such requests about addresses that are not in its list of active leases or static assignments. Hence the need for such.

  2. That pings go through I don't have an explanation for, but ICMP is a distinct protocol from ARP or TCP and may get effectively broadcast across all the router's interfaces.

12
  • Sweet, I'm gonna add the static addresses in the router as well and see what happens. I've been switching off & on #3 to fix this issue for a couple of months! No idea what could be going on, but #3 shouldn't interact with the other two at all (in regard to the DOS suggestion).
    – AlvaroP
    Commented Feb 27, 2023 at 18:44
  • Sadly, this didn't work. More info: when the network freezes, I need to power cycle #3 for all RPis to respond to ssh again. I tried power cycling the others and, even when they booted up again, I couldn't ssh any of them, but I could access the web services in #1 and #2. Puzzling.
    – AlvaroP
    Commented Feb 28, 2023 at 20:35
  • What I mean by frozen ssh is that the terminal gets stuck, doesn't explicitly timeout or print anything, and doesn't respond to the usual escape sequence. I have to quit the terminal by closing it.
    – AlvaroP
    Commented Feb 28, 2023 at 20:38
  • What happens when you try to connect againt with ssh's verbosity increased (-v = some, -vv= more)?. Wireshark might be very useful here although that could require some previous experience. Also would be good if you could hotplug a screen and keyboard to see what's going on when the freeze happens.
    – goldilocks
    Commented Mar 1, 2023 at 14:48
  • This is the output of ssh -vvv pi@<RPi #3's IP>: pastebin.com/swjNaNy9. It gets stuck there forever until I quit the terminal. (Note that the RPis are set up to accept ssh connections with ssh key only, without password prompting.) Debug logs are similar for the other RPi's as well.
    – AlvaroP
    Commented Mar 2, 2023 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.