5

The Issue:

SSH connections from a Macbook (Catalina - the SSH Client) to an RPi 4B (Buster, Linux v 4.19.97-v7l+ - the SSH Server) are reliably dropped at random times, but SSH connections from the same Macbook to an Ubuntu Server (18.04 LTS - another SSH Server) are never (hardly ever) dropped. For this reason, I think the cause is the RPi, and not the Macbook, but I am keeping an open mind. I am trying to understand the cause for this, but the primary question here is how to prevent the SSH disconnects.

  • The error message is always: client_loop: send disconnect: Broken pipe

  • The disconnects happen when the Macbook's lid is open, and when it's closed.

  • The disconnects happen with 3B and 1BP RPis also, but my question is only for the 4B.

Here's the setup:

1. /etc/ssh/sshd_config for Ubuntu & RPi are virtually identical; diffs are:

For the Ubuntu Server:
# $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $
Line 96: #UseLogin no
Line 123: PasswordAuthentication yes

For RPi 4B, sshd_config is default (as I found it) :
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

2. Login/Authentication:

  • Ubuntu logins are done w/ userid & password;
  • RPi logins are done w/ stored SSH credentials.

3. Network Connection & Provisioning:

  • Both RPI and Ubuntu are connected to the LAN via Ethernet.
  • Ubuntu Server has a fixed IP, RPi 4B uses DHCP.
  • Macbook connects to the same LAN as the servers, uses same DHCP server, but is connected via WiFi

4. SSH versions:

  • Macbook:

    % ssh -V
    OpenSSH_8.1p1, LibreSSL 2.7.3
    
  • Ubuntu:

    $ ssh -V
    OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
    
  • Raspberry Pi 4B:

    $ ssh -V  
    OpenSSH_7.9p1 Raspbian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019  
    

5. Keep-Alive Arrangements:

I've tried numerous settings on both the client-side (ssh), and the server-side (sshd). The results have been the same: RPi connections reliably drop at random times, Ubuntu Server connections never (hardly ever) drop. I finally decided to use default /etc/ssh/sshd_config files on all SSH servers, and deal with Keep-Alive in the (Macbook) SSH client. Here's what I've got now:

% cat .ssh/config
Host *
    UseKeychain yes
    AddKeysToAgent yes
    ServerAliveInterval 120

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.

As of 2021-04-25, the Mac's client configuration at ~/.ssh/config is:

Host *
    UseKeychain yes
    AddKeysToAgent yes
    ServerAliveInterval 6000
    ServerAliveCountMax 240
    TCPKeepAlive no

6. Is it a Power Issue?

I don't think it's a power issue as discussed here for three (3) reasons:

  • I'm using a 5V, 3A USB-C Power Supply mfd by Apple w/ a non-eMarked cable.

  • The Power Supply is plugged into a massive UPS that's lightly loaded. The UPS log shows no issues for weeks on end.

  • I have 2 other Raspberry Pis: a 1BP (buster), and a 3B (stretch) that reliably but randomly drop SSH connections just as the 4B does.

7. Is it a Macbook Issue?

Obviously I don't think so, or I'd post the question somewhere else. If the Macbook sits for a while, the connection will drop

But those of you familiar with Apple's myriad schemes for power saving may suspect that is the most likely cause. I thought so too initially, but as I've experimented w/ this, I've come to feel it's more likely a Raspberry Pi issue. ICYI, a s/s of the Macbook (SSH Client) Power Management settings is shown below. There's also a power management mode called standby that has been disabled (i.e. never go into standby) and confirmed as follows:

% sudo pmset -a standby 0 
% pmset -g | grep standby
 standbydelaylow      10800
 standby              0          # no standby confirmed
 standbydelayhigh     86400
 highstandbythreshold 50

Energy Saver config - macOS Catalina

16
  • 2
    I'm just looking at a similar issue with my RasPi using Buster. It does not disconnect completely but after a time it does not prompt immediately when hitting enter. It takes some time. Maybe it's the same reason? Try to decrease ServerAliveInterval maybe down to 15. Test ssh from the Ubuntu server to the RasPi.
    – Ingo
    Apr 12 '20 at 11:02
  • 1
    From my Mac (Mojave) I get the same error after the Mac goes to sleep + I see the WiFi connection acting oddly - it can reach internal addresses but not any external ones! It's possible the System Settings / Energy Saver / Power Nap needs ticking. I'm just building a 64Bit x86 linux box and will try this in the next day or so...
    – user115418
    Apr 12 '20 at 11:07
  • 1
    @Ingo: I'll try changing the ServerAliveInterval. Wrt ssh from Ubuntu to RPi: I did that. It works, but when RPi disconnects, so does Ubuntu connection. I think what I need to properly test is two connections to Ubuntu: one to RPi, and one just to Macbook. Any thoughts on a method to log these "Keep Alive" messages? I don't even know the source of the disconnect error message... Server or Client?
    – Seamus
    Apr 12 '20 at 21:36
  • 1
    @Andyroo: Yeah - I'm pretty sure that Sleep = No WiFi. I've played around with the power settings in the GUI, but nothing seems to help. There are LOTS more options in pmset, but documentation is poor. In fact, it's difficult to tell what happens in any of these modes - seems Apple changes them at their whim & can't be bothered to document anything. (Yes, I am currently unhappy with several of Apple's decisions.)
    – Seamus
    Apr 12 '20 at 21:41
  • 1
    I would try with ssh -vvv -o ServerAliveInterval=30 pi@<raspi-ip>. There you have extended logging (-vvv) and a simple way to fiddle with ServerAliveInterval. I think the disconnect error message is from the client when the server cuts the session.
    – Ingo
    Apr 12 '20 at 23:37
2

Sorry to say I think this is a Mac issue and not a Pi issue.

I set up this for a test yesterday:

Pi <==> Ethernet to Router <==> Mains network <==> Airport 5Ghz <==> Mac terminal

x86 Ubuntu server Ethernet <==> Mac terminal (i.e. directly connected)

Running ifconfig on the Pi box gives me:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.10  netmask 255.255.255.0  broadcast 192.168.1.255
        ether b8:27:eb:15:ce:cf  txqueuelen 1000  (Ethernet)

Running ifconfig on the Ubuntu box gives me:

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.2.3  netmask 255.255.255.0  broadcast 192.168.2.255
    inet6 fe80::230:1bff:febf:eca4  prefixlen 64  scopeid 0x20<link>

Note the PI is IPv4 only, where as the Ubuntu box is IPv4 and IPv6

Connection to Ubuntu via IPv6 - Connection to PI via IPv4 - Internet sharing set on for Ubuntu link to stop the Mac going to sleep though screen saver set for 10 mins and sleep set for 45 mins. Sharing Internet overrides sleep mode.

Pi not doing anything but sitting at the command line Ubuntu copying nearly 3TB of data (1000s of files) via rsync -v - so lots and lots of text coming out on to the terminal session

Came back just now (just got up) to the Ubuntu linked shell showing:

packet_write_wait: Connection to fe80::230:1bff:febf:eca4%bridge100 port 22: Broken pipe

So the Mac is reporting the drop to the x86 box while the Pi connection was still solid and active.

So it is a possible overrun issue somewhere in the Terminal program / shell or network stack on the Apple - I often see the fail after lots and lots of text has been output (I'm moving thousands of archived pictures, files and TimeMachine backups around at the mo sorting out old 2014-2016 archives).

I have now seen this on both IPv6 and IPv4 connections BUT not seen it in remote TimeMachine backups.

A very quick trawl though the Mac system logs only shows Terminal starting so that's not a lot of help (TBH I'm not great at Mac debugging - not a lot of issues on it so could have missed something.)

2
  • Ran the same task overnight to just now again but with output going to a disconnected tmux session - both the Pi and the Ubuntu box stayed up fine. Convinced its a Mac issue
    – user115418
    Apr 14 '20 at 14:53
  • Thanks for your answer. I now agree with your opening assertion, "I think this is a Mac issue and not a Pi issue" (+1), BUT: I don't feel the rest of your answer re causation is totally relevant. I've discovered some information and seen enough now to convince me that your opening assertion is correct, but "the answer" seems to lie in some arcane settings in macOS. I'll post something like an answer within the next day or two - even though this now seems "off-topic" for Raspberry Pi SE.
    – Seamus
    Apr 14 '20 at 20:27
1

I never made much progress trying to chase this down to determine the "root cause". However, for reasons unknown to me, the number of dropped SSH connections has decreased to near zero. The SSH software hasn't changed since I posted the question about 4 months ago... as of today:

RPi4B:

$ ssh -V
OpenSSH_7.9p1 Raspbian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019

macOS:

% ssh -V                                                        
OpenSSH_8.1p1, LibreSSL 2.7.3

Of course there have been updates to both systems since the question was posted. Perhaps something in the networking stacks of one or both OS? At this point it seems unlikely a definitive answer will emerge, but I'll leave this open for a while longer before "closing" it with acceptance of an answer.

1
  • Update: Since posting this "non-answer" reporting the frequency of dropped connections had abated, the trend reversed itself, and SSH connections drop reliably. I've tried launching the SSH connections under Mac's caffeinate, but it seems to have no effect. :(
    – Seamus
    Apr 24 at 21:54
1

This issue has persisted with Big Sur. In hunting around, the solution may be to add/edit to your ~/.ssh/config file on your Mac the following:

TCPKeepAlive no

The reasoning for and source of this proposed answer can be found here: https://unix.stackexchange.com/questions/602518/ssh-connection-client-loop-send-disconnect-broken-pipe-or-connection-reset

Quoting from that source:

TCPKeepAlive no means "do not send keepalive messages to the server". When the opposite, TCPKeepAlive yes, is set, then the client sends keepalive messages to the server and requires a response in order to maintain its end of the connection. This will detect if the server goes down, reboots, etc. The trouble with this is that if the connection between the client and server is broken for a brief period of time (due to flaky a network connection), this will cause the keepalive messages to fail, and the client will end the connection with "broken pipe".

Setting TCPKeepAlive no tells the client to just assume the connection is still good until proven otherwise by a user request, meaning that temporary connection breakages while your ssh term is sitting idle in the background won't kill the connection.

5
  • The explanation sounds reasonable (+1), and definitely worth a try. I've just added a comment confirming my "non-answer" hasn't actually made any difference. I'll let you know if your suggestion helps. And BTW - you are still very much in the running for "The Accepted Answer", but you should structure your answer in the form of a recommended ~/.ssh/config file, and indicate what your results have been.
    – Seamus
    Apr 24 at 22:05
  • Yeah, I'll try to do so tomorrow. I'm new posting here and sorting out the editing features, but got a bit busy with life. Considering I can't even figure out how to insert a newline in this shit comment, who knows. Neither command, option, or control + enter works. Fuck me. Anyhow, since it's not really my answer but the original answerer's, is there an accepted way for me to structure the answer such that it better points to his, and he gets the credit...if it works?
    – MikeSilva
    Apr 26 at 11:44
  • LOL... yeah, the "comment box" uses a different set of Markdown rules - it kinda' sucks :) As far as the "original answerer": don't worry about him - he no longer has an account here (note the pale user handle & no 'popup' when hovering over it). He may have been correct that it's a Mac issue, BUT the connection drops even when I launch SSH connection under caffeinate which according to Apple prevents sleep. Do you use a Mac?
    – Seamus
    Apr 26 at 21:57
  • Oh - one other thing re the comment boxes: Notifications. When I post a comment to your answer, you're notified because it's your post. However, if you reply to my comment, I may not get a notification unless you put my handle at the beginning of your reply; i.e. "@seamus blah, blah...". I think this is documented somewhere, but can't find it now.
    – Seamus
    Apr 26 at 22:04
  • Ooops... better late than never, I hope. To finally follow up, I've made the change you suggested, and tested this on my Mac (Catalina 10.15.6). Unfortunately, it does not cure the problem.
    – Seamus
    May 28 at 21:46
0

My solution steps to reproduce:

  1. be sure you are connected on internet so download and install Raspberry Pi Imager from official https://www.raspberrypi.org/software/
  2. so from your macOS open up Raspberry Pi Imager
  3. at operating system choose Erase Format card as FAT32. So choose Storage and then Write
  4. at operating system choose Raspberry Pi OS (other) then choose Raspberry Pi OS Lite (32-bit). So choose Storage and then Write
  5. to let ssh available into raspberrypi just create an empty file called ssh into /Volumes/boot
  6. you can now safely disconnect /Volumes/boot
  7. power-on via ethernet this raspberrypi with the new boot
  8. be sure on your macOS file ~/.ssh/config is empty
  9. be sure on your macOS file ~/.ssh/known_hosts is empty
  10. from your macOS open up a terminal and run command ssh pi@raspberrypi.local so choose yes and type default password raspberry
  11. get ip of raspberrypi by running command hostname -I
  12. run command exit
  13. now use command ssh pi@<ip address you copied just above> so choose yes and type default password raspberry
  14. If still connections drops factory reset your router

Environment

macOS
➜  ~ sw_vers        
ProductName:    macOS
ProductVersion: 11.4
BuildVersion:   20F71

➜  ~ /usr/bin/ssh -V
OpenSSH_8.1p1, LibreSSL 2.7.3
raspberrypi
pi@raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

pi@raspberrypi:~ $ ssh -V
OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1, OpenSSL 1.1.1d  10 Sep 2019

Note

  • Default /etc/ssh/sshd_config looks like this pastebin
9
  • What settings did you have in the ~/.ssh/config file on your Mac?
    – Seamus
    Jun 9 at 7:18
  • I have tried using the IP address instead of the .local address - it made no difference. Your comment re "coherent" is a bit ambiguous to me - do you mean that the settings must be the same?... which ones exactly? I appreciate your answer, but your lack of any detail isn't particularly enlightening... IOW - how did you test your solution?
    – Seamus
    Jun 13 at 0:27
  • I'll try this, but I have a question for you first: Is your local machine a Mac? Windows? Linux?
    – Seamus
    Jun 14 at 19:26
  • OK, thanks - I really appreciate your effort. I'll try this again, but please note that my question is about the Mac-to-RPi SSH connection - specifically mac os Catalina. It's interesting that the same ver of SSH and LibreSSL is used on Catalina & Mojave.
    – Seamus
    Jun 15 at 2:51
  • My mistake - Big Sur, not Mojave. But your statement that the two environments (Catalina & Big Sur) are the exact same is a bit of a stretch - there are more variables due to diffs in the other software components. But let's not get distracted - I said I'll try your suggestion, and I will. And if that fixes my issue, then your answer will become the "accepted answer"
    – Seamus
    Jun 15 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.