Has anyone solved Raspberry Pi Zero W SSH: `client_loop: send disconnect: broken pipe/timeout/Connection reset`

Question

I know this has been voiced before, but the most recent I've found was in 2020 and it wasn't resolved then, so I'm signal-boosting in the hope that an answer has been discovered and just wasn't posted...

I have a group of headless Pi Zero W's that I was setting up as IR transceivers. They were working fine a year ago when I set that project aside for a while. They're configured to come up at specific network addresses, and I was able to ssh into them with no problems.

Just started trying to pick up that project again, and Something Is Very Wrong.

I can ping these systems with no problem, so they're booting up and running.

Trying to SSH into them, however, I either get through the login to the command prompt and then the connection becomes unresponsive... or before it gets all the way through the login the connection falls on its face.

Example of the latter:

C:\Users\keshlam>ssh keshlamIR@ir3
Linux ir3 5.15.84+ #1613 Thu Jan 5 11:58:09 GMT 2023 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon May 15 02:17:40 2023
client_loop: send disconnect: Connection reset

C:\Users\keshlam>

Again, these Pis have been running-but-unattended for the past year. I haven't changed the router/firewall configurations. Ping still claims they're alive. Power-cycling them doesn't improve matters.

Any ideas on what kind of bit decay I've suffered and how to get these back into operation (and prevent a recurrence) would be greatly appreciated. The Pi is an amazing little piece of equipment, when it works.

Arggh?

ADDITIONAL INFO:

I can work with the pi via keyboard/console, so it's booting up.

I ran a full apt update/upgrade and rebooted.

I'm not out of space on the SD cards, by a long shot.

When I don't get the above error I instead see what looks like successful login but a hang before the shell responds:

C:\Users\keshlam>ssh keshlamIR@ir3
Linux ir3 6.1.21+ #1642 Mon Apr  3 17:19:14 BST 2023 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon May 15 17:51:17 2023

(and it usually stops there. Occasionally I'll get the shell
prompt and then it freezes, or freezes after the first
character or two has been typed at the prompt.)

ssh_config did not explicitly say PermitTTY yes, but adding that d rebooting didn't let my ssh reach the shell prompt.

Running ssh -Y (needed in the past for X apps to reach back to my Windows X server) makes no apparent difference.

When I ssh -vvvv, after authentication I'm seeing

debug1: Authentication succeeded (publickey).
Authenticated to ir3 ([192.170.3.13]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [email protected]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug1: ENABLE_VIRTUAL_TERMINAL_INPUT is supported. Reading the VTSequence from console
debug3: This windows OS supports conpty
debug1: ENABLE_VIRTUAL_TERMINAL_PROCESSING is supported. Console supports the ansi parsing
debug3: Successfully set console output code page from:65001 to 65001
debug3: Successfully set console input code page from:65001 to 65001
debug3: receive packet: type 80
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug3: receive packet: type 4
debug1: Remote: /home/keshlamIR/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /home/keshlamIR/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768

(and it stops there)

... which looks OK as far as it goes?

I did sometimes see a surprisingly long pause between authentication and the shell prompt when logging in a year ago -- but that was on the order of seconds, not the extended stop I'm seeing now.

I'm still hoping this is something obvious. I really need to get this project rolling again. And given the current supply chain, swapping out the RPis for something else would be not only painful but expensive.

Answer 1

I've had problems with this for years, but my SSH clients are all Apple Mac devices. I've improved my results by using the caffeinate command on macOS, but of course that doesn't help you if you're using a Windows client.

I think that the problem is at least partially caused by "sleeping" or low power states on the client - energy conservation measures. I've not done extensive testing, but it seems that Pi-to-Pi SSH connections don't have this problem (with dropped connections).

Another trick I've used with some success that may help with your Windows client connections is using this command:

$ ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=10 [email protected]

This effectively causes "keep alives" to be exchanged between client and server. If Windows has something like caffeinate that allows you to designate certain processes as exempt from sleep, you may want to try that.

Answer 2

Found the solution. Or at least a solution.

Apparently, open-ssh has relatively recently begun using quality-of-service (QoS) metrics to try to optimize its performance based on the expected behavior of the connection, and its default is to assume a reliability that the Pi Zero W's tiny wifi simply can't deliver; as a result it gets caught waiting for lost packets rather than retrying.

Theoretically, adding the option -o IPQoS=0 to the ssh command ought to override that default for this session. I haven't had much success with that approach.

What did work for me was connecting keyboard and display to the Pi and editing /etc/ssh/ssh_conf and /etc/ssh/sshd_conf to add a line setting this value:

IPQoS 0x00

There are other values which may be worth trying. I don't yet understand the exact details of what each does to adjust the protocols, so I don't have any opinions or suggestions on which. But setting it to zero does seem to get my pi0w's back on the air.

Might be nice if there was a Pi setup tool which looked at what hardware it was actually running on and tuned this sort of thing. Oh well.