What causes Raspberry Pi to reliably drop SSH connections: `client_loop: send disconnect: Broken pipe`

Question

The Issue:

SSH connections from a Macbook (Catalina - the SSH Client) to an RPi 4B (Buster, Linux v 4.19.97-v7l+ - the SSH Server) are reliably dropped at random times, but SSH connections from the same Macbook to an Ubuntu Server (18.04 LTS - another SSH Server) are never (hardly ever) dropped. For this reason, I think the cause is the RPi, and not the Macbook, but I am keeping an open mind. I am trying to understand the cause for this, but the primary question here is how to prevent the SSH disconnects.

• The error message is always: client_loop: send disconnect: Broken pipe

• The disconnects happen when the Macbook's lid is open, and when it's closed.

• The disconnects happen with 3B and 1BP RPis also, but my question is only for the 4B.

Here's the setup:

1. /etc/ssh/sshd_config for Ubuntu & RPi are virtually identical; diffs are:

For the Ubuntu Server:
# $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $
Line 96: #UseLogin no
Line 123: PasswordAuthentication yes

For RPi 4B, sshd_config is default (as I found it) :
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

2. Login/Authentication:

• Ubuntu logins are done w/ userid & password;
• RPi logins are done w/ stored SSH credentials.

3. Network Connection & Provisioning:

• Both RPI and Ubuntu are connected to the LAN via Ethernet.
• Ubuntu Server has a fixed IP, RPi 4B uses DHCP.
• Macbook connects to the same LAN as the servers, uses same DHCP server, but is connected via WiFi

4. SSH versions:

• Macbook:

  % ssh -V
  OpenSSH_8.1p1, LibreSSL 2.7.3
  

• Ubuntu:

  $ ssh -V
  OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
  

• Raspberry Pi 4B:

  $ ssh -V  
  OpenSSH_7.9p1 Raspbian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019  
  

5. Keep-Alive Arrangements:

I've tried numerous settings on both the client-side (ssh), and the server-side (sshd). The results have been the same: RPi connections reliably drop at random times, Ubuntu Server connections never (hardly ever) drop. I finally decided to use default /etc/ssh/sshd_config files on all SSH servers, and deal with Keep-Alive in the (Macbook) SSH client. Here's what I've got now:

% cat .ssh/config
Host *
    UseKeychain yes
    AddKeysToAgent yes
    ServerAliveInterval 120

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.

As of 2021-04-25, the Mac's client configuration at ~/.ssh/config is:

Host *
    UseKeychain yes
    AddKeysToAgent yes
    ServerAliveInterval 6000
    ServerAliveCountMax 240
    TCPKeepAlive no

6. Is it a Power Issue?

I don't think it's a power issue as discussed here for three (3) reasons:

• I'm using a 5V, 3A USB-C Power Supply mfd by Apple w/ a non-eMarked cable.

• The Power Supply is plugged into a massive UPS that's lightly loaded. The UPS log shows no issues for weeks on end.

• I have 2 other Raspberry Pis: a 1BP (buster), and a 3B (stretch) that reliably but randomly drop SSH connections just as the 4B does.

7. Is it a Macbook Issue?

Obviously I don't think so, or I'd post the question somewhere else. If the Macbook sits for a while, the connection will drop

But those of you familiar with Apple's myriad schemes for power saving may suspect that is the most likely cause. I thought so too initially, but as I've experimented w/ this, I've come to feel it's more likely a Raspberry Pi issue. ICYI, a s/s of the Macbook (SSH Client) Power Management settings is shown below. There's also a power management mode called standby that has been disabled (i.e. never go into standby) and confirmed as follows:

% sudo pmset -a standby 0 
% pmset -g | grep standby
 standbydelaylow      10800
 standby              0          # no standby confirmed
 standbydelayhigh     86400
 highstandbythreshold 50

Answer 1

Sorry to say I think this is a Mac issue and not a Pi issue.

I set up this for a test yesterday:

Pi <==> Ethernet to Router <==> Mains network <==> Airport 5Ghz <==> Mac terminal

x86 Ubuntu server Ethernet <==> Mac terminal (i.e. directly connected)

Running ifconfig on the Pi box gives me:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
        ether b8:27:eb:15:ce:cf  txqueuelen 1000  (Ethernet)

Running ifconfig on the Ubuntu box gives me:

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.2.3  netmask 255.255.255.0  broadcast 192.168.2.255
    inet6 fe80::230:1bff:febf:eca4  prefixlen 64  scopeid 0x20<link>

Note the PI is IPv4 only, where as the Ubuntu box is IPv4 and IPv6

Connection to Ubuntu via IPv6 - Connection to PI via IPv4 - Internet sharing set on for Ubuntu link to stop the Mac going to sleep though screen saver set for 10 mins and sleep set for 45 mins. Sharing Internet overrides sleep mode.

Pi not doing anything but sitting at the command line Ubuntu copying nearly 3TB of data (1000s of files) via rsync -v - so lots and lots of text coming out on to the terminal session

Came back just now (just got up) to the Ubuntu linked shell showing:

packet_write_wait: Connection to fe80::230:1bff:febf:eca4%bridge100 port 22: Broken pipe

So the Mac is reporting the drop to the x86 box while the Pi connection was still solid and active.

So it is a possible overrun issue somewhere in the Terminal program / shell or network stack on the Apple - I often see the fail after lots and lots of text has been output (I'm moving thousands of archived pictures, files and TimeMachine backups around at the mo sorting out old 2014-2016 archives).

I have now seen this on both IPv6 and IPv4 connections BUT not seen it in remote TimeMachine backups.

A very quick trawl though the Mac system logs only shows Terminal starting so that's not a lot of help (TBH I'm not great at Mac debugging - not a lot of issues on it so could have missed something.)

Answer 2

This issue has persisted with Big Sur. In hunting around, the solution may be to add/edit to your ~/.ssh/config file on your Mac the following:

TCPKeepAlive no

The reasoning for and source of this proposed answer can be found here: https://unix.stackexchange.com/questions/602518/ssh-connection-client-loop-send-disconnect-broken-pipe-or-connection-reset

Quoting from that source:

TCPKeepAlive no means "do not send keepalive messages to the server". When the opposite, TCPKeepAlive yes, is set, then the client sends keepalive messages to the server and requires a response in order to maintain its end of the connection. This will detect if the server goes down, reboots, etc. The trouble with this is that if the connection between the client and server is broken for a brief period of time (due to flaky a network connection), this will cause the keepalive messages to fail, and the client will end the connection with "broken pipe".

Setting TCPKeepAlive no tells the client to just assume the connection is still good until proven otherwise by a user request, meaning that temporary connection breakages while your ssh term is sitting idle in the background won't kill the connection.

Answer 3

I never made much progress trying to chase this down to determine the "root cause". However, for reasons unknown to me, the number of dropped SSH connections has decreased to near zero. The SSH software hasn't changed since I posted the question about 4 months ago... as of today:

RPi4B:

$ ssh -V
OpenSSH_7.9p1 Raspbian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019

macOS:

% ssh -V                                                        
OpenSSH_8.1p1, LibreSSL 2.7.3

Of course there have been updates to both systems since the question was posted. Perhaps something in the networking stacks of one or both OS? At this point it seems unlikely a definitive answer will emerge, but I'll leave this open for a while longer before "closing" it with acceptance of an answer.

Answer 4

In my case it turned out to be a Router issue. I was using dd-wrt and had to disable "Shortcut Forwarding Engine" in the Setup page.

Answer 5

I would just like to add that I experience a similar issue still on a rPI 4B with rasbian/debian 11 (bullseye).

My setup is a Rpi connected to ethernet (via powerline) then to a router/modem. And my laptop (debian testing / bookworm) via wifi to the same router.

SSH sessions randomly close with an error

client_loop: send disconnect: Broken pipe

Also many times I have to try 2 or 3 times to connect on the rpi, since I get this error without even getting a terminal

Connection reset by 192.168.2.130 port 22

As a workaroung I also installed tellnetd on the rpi and honestly it feels strange to use telnet after all, but it works ! I mean I have simultaneous connections from my laptop to rpi with ssh and telnet, and while the first one dies randomly with client loop... the other is not affected!

I used to have the debian 11 ssh version 1:8.4p1-5+deb11u1 still having the issue. Then upgraded to testing version 1:9.0p1-1+b2 and still have the issue. So for server it is

sshd -V OpenSSH_9.0p1 Debian-1+b2, OpenSSL 3.0.5 5 Jul 2022

and for the client

ssh -V OpenSSH_9.0p1 Debian-1+b2, OpenSSL 3.0.5 5 Jul 2022

SSH versions on server(rpi)

I have also tried to disable TCPKeepAlive both on client and server to no avail.

edit It seems that the problem is caused by the router somehow emiting RST packets in one way of the TCP stream, thus breaking the connection. It this case I could verify that the other end (on the rPI ssh server) has still an ESTABLISHED tcp connection that does not exist on the client side.

Answer 6

My solution steps to reproduce:

1. be sure you are connected on internet so download and install Raspberry Pi Imager from official https://www.raspberrypi.org/software/
2. so from your macOS open up Raspberry Pi Imager
3. at operating system choose Erase Format card as FAT32. So choose Storage and then Write
4. at operating system choose Raspberry Pi OS (other) then choose Raspberry Pi OS Lite (32-bit). So choose Storage and then Write
5. to let ssh available into raspberrypi just create an empty file called ssh into /Volumes/boot
6. you can now safely disconnect /Volumes/boot
7. power-on via ethernet this raspberrypi with the new boot
8. be sure on your macOS file ~/.ssh/config is empty
9. be sure on your macOS file ~/.ssh/known_hosts is empty
10. from your macOS open up a terminal and run command ssh [email protected] so choose yes and type default password raspberry
11. get ip of raspberrypi by running command hostname -I
12. run command exit
13. now use command ssh pi@<ip address you copied just above> so choose yes and type default password raspberry
14. If still connections drops factory reset your router

Environment

macOS

➜  ~ sw_vers        
ProductName:    macOS
ProductVersion: 11.4
BuildVersion:   20F71

➜  ~ /usr/bin/ssh -V
OpenSSH_8.1p1, LibreSSL 2.7.3

raspberrypi

pi@raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

pi@raspberrypi:~ $ ssh -V
OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1, OpenSSL 1.1.1d  10 Sep 2019

Note

• Default /etc/ssh/sshd_config looks like this pastebin