Python Script with Systemctl not working only on Stretch

Question

I've been using systemctl to manage multiple python scripts on raspberry pi 3b running jessie for a while. I recently got a rpi 3b+ and installed the lates Stretch on it. Set up the exact same way as the older one but can't get the services working properly. Below is the service and the python script i am testing just now.

Service:

[Unit]
Description=Keep Alive Service
After=multi-user.target

[Service]
Type=idle
ExecStart=/usr/bin/python /home/pi/Desktop/keep_alive.py > /home/pi/Desktop/keep_alive.log 2>&1
Restart=always

[Install]
WantedBy=multi-user.target

Python Script:

import time
import os

while 1:
    command = 'sshpass -p "server_pass" ssh -N -o ExitOnForwardFailure=yes -R port:localhost:22 server_username@server_ip_address'
    os.system(command)
    time.sleep(60)

This should create a reverse ssh tunnel to a server to allow me for remote login to the pi.

Now if i enable and start the service, then check its status i get the following:

● keep_alive_service.service - Keep Alive Service
   Loaded: loaded (/lib/systemd/system/keep_alive_service.service; enabled; vend
   Active: active (running) since Fri 2018-06-15 09:44:08 BST; 12min ago
 Main PID: 24289 (python)
   CGroup: /system.slice/keep_alive_service.service
           └─24289 /usr/bin/python /home/pi/Desktop/keep_alive.py > /home/pi/Desktop/keep_alive.log 2>&1

Jun 15 09:44:08 M-HAS systemd[1]: Started Keep Alive Service.

And also checking with ps -aef | grep python i see the script is running:

root     24289     1  0 09:44 ?        00:00:00 /usr/bin/python /home/pi/Desktop/keep_alive.py > /home/pi/Desktop/keep_alive.log 2>&1

However am sure the tunnel is not created. If i try to log in from the server i get connection refused. If i manually run the script it works fine.

EDIT 1

So i tried a couple more things to udnerstand how the python script is run by systemctl (since it shows as running). First of all i tried:

• stop the service.
• disconnect the pi form the internet.
• restart the service.

Checking the status and logs i can see (as expected) an error "Network is unreachable" when the ssh command is executed, and then again every 60 seconds after since the loop keeps going.

Now, i reconnected the pi to the internet added a print line just after the ssh command within the script as follows:

import time
import os

while 1:
    command = 'sshpass -p "server_pass" ssh -N -o ExitOnForwardFailure=yes -R port:localhost:22 server_username@server_ip_address'
    os.system(command)
    print 'got here'
    time.sleep(60)

So again, if i run the script manually, all works well and that print never gets printed out since python doesn't get past the os.system().

However if i run the script with systemctl it seems that the os.system() somehow fails, and 'got here' is printed. The thing is, there is no actual errors shown, neither in the log or when checking the status of the service. This is way i tested without internet connection earlier to see what would happend, and did get "Network is unreachable". This just confused me even more.

Also this whole set up works fine on a rpi 3b running jessie.

Is there some difference between jessie / stretch and or rpi 3b and rpi 3b+ when using systemctl? I cant figure out what is wrong.

Answer 1

Raspbian Jessie comes with systemd 215, Raspbian Stretch uses systemd 232. Differences between these versions you can see in systemd NEWS. There are very much changes and reflects heavy development. It's not surprising that Stretch behaves others than Jessie.

With your stripped down test python script is something wrong. It will only set in an endless loop (while 1:) the variable command and never executes os.system(command). I don't understand why do you see it running with ps -aef. If you also indent os.system(command) and time.sleep(60) into the loop, it will create a new ssh connection every hour. After one day you have 24 connections.

In the Unit file I would set Type=simple instead of Type=idle. man systemd.service says to idle:

Note that this type is useful only to improve console output, it is not useful as a general unit ordering tool, and the effect of this service type is subject to a 5s time-out, after which the service binary is invoked anyway.

If you can invoke the script on the command line it is unclear if you run it as a user or as root with sudo. If you run it as user you should set it in the unit file.

I don't know if it is a good idea to Restart=always. It may be that it slow down your whole system by attempting to restart the service without delay, e.g. when the network connection is down.

To put it all together I would set it up with optimized for network online as follows:

[Unit]
Description=Keep Alive Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
# if needed uncomment next line
#User=pi
ExecStart=/usr/bin/python /home/pi/Desktop/keep_alive.py > /home/pi/Desktop/keep_alive.log 2>&1

[Install]
WantedBy=multi-user.target

To monitor a ssh connection it is better to use autossh. It is just for this.