1

I have a Raspberry Pi Zero W (PiZeroW-1, Raspbian Buster)on a local network behind a 4G router with CGNat. To be able to communicate with it from elsewhere I have setup a reverse ssh tunnel that is launched automatically after booting by systemd, by means of a ssh server on a second raspberry pi (PiZero2-1, Raspbian Bullseye) that can be accessed externally through port 22 forwarded to port 2222 on the second router:

/etc/systemd/system/ssh_reverse_tunnel.service:

[Unit]
Description=reverse ssh tunnel service
After=multi-user.target
Requires=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=on-failure
RestartSec=1
User=pi
ExecStart=/usr/bin/ssh -tt -R 19999:localhost:22 pi@xxxx.ddns.net -p 2222
StartLimitIntervalSec=0
[Install]
WantedBy=multi-user.target

After booting I can access from the second Raspi the one behind the 4G router without any issue by means of:

pi@PiZero2-1:~ $ ssh localhost -p 19999

After a few hours (sometimes a few days), the tunnel stops working from the second Raspi side:

pi@PiZero2-1:~ $ ssh localhost -p 19999
ssh: connect to host localhost port 19999: Connection refused

When this happens, the tunnel is still working from the first Raspi side:

pi@PiZeroW-1:~ $ systemctl status ssh_reverse_tunnel

● ssh_reverse_tunnel.service - reverse ssh tunnel service
   Loaded: loaded (/etc/systemd/system/ssh_reverse_tunnel.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-12-20 18:01:41 CET; 1min 8s ago
 Main PID: 597 (ssh)
    Tasks: 1 (limit: 725)
   CGroup: /system.slice/ssh_reverse_tunnel.service
           └─597 /usr/bin/ssh -tt -R 19999:localhost:22 pi@xxxx.ddns.net -p 2222

dic 20 18:01:41 PiZeroW-1 systemd[1]: Started reverse ssh tunnel service.
dic 20 18:02:04 PiZeroW-1 ssh[597]: Linux PiZero2-1 5.10.63+ #1488 Thu Nov 18 16:14:04 GMT 2021 armv6l
dic 20 18:02:04 PiZeroW-1 ssh[597]: [1B blob data]
dic 20 18:02:04 PiZeroW-1 ssh[597]: The programs included with the Debian GNU/Linux system are free software;
dic 20 18:02:04 PiZeroW-1 ssh[597]: the exact distribution terms for each program are described in the
dic 20 18:02:04 PiZeroW-1 ssh[597]: individual files in /usr/share/doc/*/copyright.
dic 20 18:02:04 PiZeroW-1 ssh[597]: [1B blob data]
dic 20 18:02:04 PiZeroW-1 ssh[597]: Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
dic 20 18:02:04 PiZeroW-1 ssh[597]: permitted by applicable law.
dic 20 18:02:04 PiZeroW-1 ssh[597]: Last login: Mon Dec 20 17:54:58 2021 from 95.127.169.53

pi@PiZeroW-1:~ $ ps -ax | grep ssh

  426 ?        Ss     0:00 /usr/sbin/sshd -D
  619 ?        Ss     0:01 /usr/bin/ssh -tt -R 19999:localhost:22 pi@xxxx.ddns.net -p 2222
  639 ?        Ss     0:00 sshd: pi [priv]
  669 ?        S      0:01 sshd: pi@pts/0
  730 ?        Ss     0:00 /usr/bin/ssh-agent x-session-manager
  801 ?        Ss     0:00 /usr/bin/ssh-agent -s
 1029 ?        Ss     0:00 sshd: pi [priv]
 1035 ?        S      0:00 sshd: pi@pts/2
22544 pts/0    S+     0:00 grep --color=auto ssh

... but it is not working from the other end.

If I reboot the second Raspberry (PiZero2-1), the problem persists. If I reboot the first Raspberry (PiZeroW-1), the problems disappears (till a few hours or a few days later).

Anybody has some idea about the possible origin of the tunnel stalling ? I have been stuck here for several days and right now the only solution I can realize to guarantee to keep the communication open is to program to reboot the first Raspi from time to time...

BTW: the 4G router and its corresponding local network is to be located in my second home, 600 kms away from where I normally live. I must guarantee that once started the communication will not stop.

0

Last night it happened again. Reverse ssh tunnel stopped working. Looking at the logs of the machine failing to connect I have seen that during the night there have been hundreds of intents of connection from an IP in India (143.244.140.96):

pi@PiZero2-1:~ $ journalctl -u ssh.service
…..
dic 20 20:22:53 PiZero2-1 sshd[2369]: pam_unix(sshd:session): session opened for user pi(uid=1000) by (uid=0)
dic 20 20:26:40 PiZero2-1 sshd[2419]: Accepted publickey for pi from 95.xxx.xxx.53 port 37231 ssh2: RSA SHA256:vqZo8JuvcYGxgVED2aUdFpedsfiH1onFw
dic 20 20:26:41 PiZero2-1 sshd[2419]: pam_unix(sshd:session): session opened for user pi(uid=1000) by (uid=0)
dic 20 20:33:48 PiZero2-1 sshd[2489]: error: kex_exchange_identification: banner line contains invalid characters
dic 20 20:33:48 PiZero2-1 sshd[2489]: banner exchange: Connection from 185.219.52.171 port 65377: invalid format
dic 20 22:27:23 PiZero2-1 sshd[3476]: error: kex_exchange_identification: Connection closed by remote host
dic 20 22:27:23 PiZero2-1 sshd[3476]: Connection closed by 143.244.140.96 port 59254
dic 20 22:27:57 PiZero2-1 sshd[3478]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=143.244.140.96  user=root
dic 20 22:27:59 PiZero2-1 sshd[3478]: Failed password for root from 143.244.140.96 port 44288 ssh2
dic 20 22:28:00 PiZero2-1 sshd[3478]: Connection closed by authenticating user root 143.244.140.96 port 44288 [preauth]
dic 20 22:28:05 PiZero2-1 sshd[3482]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=143.244.140.96  user=root
dic 20 22:28:07 PiZero2-1 sshd[3482]: Failed password for root from 143.244.140.96 port 55942 ssh2
dic 20 22:28:08 PiZero2-1 sshd[3482]: Connection closed by authenticating user root 143.244.140.96 port 55942 [preauth]
dic 20 22:28:14 PiZero2-1 sshd[3484]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=143.244.140.96  user=root
dic 20 22:28:16 PiZero2-1 sshd[3484]: Failed password for root from 143.244.140.96 port 39366 ssh2
dic 20 22:28:17 PiZero2-1 sshd[3484]: Connection closed by authenticating user root 143.244.140.96 port 39366 [preauth]
dic 20 22:28:22 PiZero2-1 sshd[3486]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=143.244.140.96  user=root
dic 20 22:28:23 PiZero2-1 sshd[3486]: Failed password for root from 143.244.140.96 port 51022 ssh2
dic 20 22:28:25 PiZero2-1 sshd[3486]: Connection closed by authenticating user root 143.244.140.96 port 51022 [preauth]
             etc…etc…
dic 20 22:49:59 PiZero2-1 sshd[3943]: Connection closed by invalid user zabbix 143.244.140.96 port 44212 [preauth]
dic 20 22:50:03 PiZero2-1 sshd[3945]: Invalid user zabbix from 143.244.140.96 port 55814
dic 20 22:50:03 PiZero2-1 sshd[3945]: pam_unix(sshd:auth): check pass; user unknown
dic 20 22:50:03 PiZero2-1 sshd[3945]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=143.244.140.96
dic 20 22:50:05 PiZero2-1 sshd[3945]: Failed password for invalid user zabbix from 143.244.140.96 port 55814 ssh2
dic 20 22:50:06 PiZero2-1 sshd[3945]: Connection closed by invalid user zabbix 143.244.140.96 port 55814 [preauth]
dic 21 01:08:15 PiZero2-1 sshd[4982]: error: kex_exchange_identification: banner line contains invalid characters
dic 21 01:08:15 PiZero2-1 sshd[4982]: banner exchange: Connection from 185.219.52.171 port 64895: invalid format
dic 21 03:24:01 PiZero2-1 sshd[6021]: error: kex_exchange_identification: Connection closed by remote host
dic 21 03:24:01 PiZero2-1 sshd[6021]: Connection closed by 185.36.81.95 port 33612
dic 21 03:24:01 PiZero2-1 sshd[6022]: Connection closed by 185.220.100.241 port 9946 [preauth]
dic 21 09:47:22 PiZero2-1 sshd[8789]: Accepted publickey for pi from 95.127.169.53 port 37122 ssh2: RSA SHA256:vqZo8JuvcYGxgVED2aUdFpedsfiH1onFw
dic 21 09:47:22 PiZero2-1 sshd[8789]: pam_unix(sshd:session): session opened for user pi(uid=1000) by (uid=0)
dic 21 09:59:09 PiZero2-1 sshd[8883]: Accepted publickey for pi from 95.127.169.53 port 37290 ssh2: RSA SHA256:vqZo8JuvcYGxgVED2aUdFpedsfiH1onFw
dic 21 09:59:09 PiZero2-1 sshd[8883]: pam_unix(sshd:session): session opened for user pi(uid=1000) by (uid=0)

pi@PiZero2-1:~ $ sudo service ssh status
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-12-20 17:43:10 CET; 16h ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 353 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 372 (sshd)
      Tasks: 1 (limit: 877)
        CPU: 59.046s
     CGroup: /system.slice/ssh.service
             └─372 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

dic 20 22:50:06 PiZero2-1 sshd[3945]: Connection closed by invalid user zabbix 143.244.140.96 port 55814 [preauth]
dic 21 01:08:15 PiZero2-1 sshd[4982]: error: kex_exchange_identification: banner line contains invalid characters
dic 21 01:08:15 PiZero2-1 sshd[4982]: banner exchange: Connection from 185.219.52.171 port 64895: invalid format
dic 21 03:24:01 PiZero2-1 sshd[6021]: error: kex_exchange_identification: Connection closed by remote host
dic 21 03:24:01 PiZero2-1 sshd[6021]: Connection closed by 185.36.81.95 port 33612
dic 21 03:24:01 PiZero2-1 sshd[6022]: Connection closed by 185.220.100.241 port 9946 [preauth]
dic 21 09:47:22 PiZero2-1 sshd[8789]: Accepted publickey for pi from 95.xxx.xxx.53 port 37122 ssh2: RSA SHA256:vqZo8JuvcYGxgVED2aUdFpedsfiH1onFw
dic 21 09:47:22 PiZero2-1 sshd[8789]: pam_unix(sshd:session): session opened for user pi(uid=1000) by (uid=0)
dic 21 09:59:09 PiZero2-1 sshd[8883]: Accepted publickey for pi from 95.xxx.xxx.53 port 37290 ssh2: RSA SHA256:vqZo8JuvcYGxgVED2aUdFpedsfiH1onFw
dic 21 09:59:09 PiZero2-1 sshd[8883]: pam_unix(sshd:session): session opened for user pi(uid=1000) by (uid=0

Events before 20:26 and after 9AM are legit logins from my side.

Could this attack explain why my tunnel stopped working ? If yes, how can I avoid the interruption of the service ?

Thanks and best regards

2
  • Brute force attempts against SSH and other Internet-facing services are a fact of life. It is doubtful that the attacks are the cause of your problem unless volume is such that it causes DDOS. Consider using software like Fail2ban, sshguard, CSF+LFD or similar to block those attacks. But you seem to be using a dynamic DNS service ?
    – Kate
    Dec 21 '21 at 15:12
  • Since you already know how to use journalctl, do the same for your ssh_reverse_tunnel.service - maybe you'll see something interesting in the logs.
    – Kate
    Dec 21 '21 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.