Python script stops mysteriously after a few hours

Question

The Problem

I have a raspberry pi 3b+ running a python 3.7 script that is controlling an arduino. The arduino is reading values in from a few different sensors, and sending the data back to the python code, where it is processed and stored. These sensors are measuring temperature, humidity, ambient light level, soil moisture, and water levels of an agroponic system, with the purpose of maintaining ideal values to facilitate plant growth. This is a timer based system, and will turn the grow light on after 16 hours to run for 8 hours and repeat, and the pump will turn on after 18 hours and run for 5 seconds (it's a strong pump) and repeat.

Obviously, plants don't grow in the span of a few hours, and this script would ideally be running for a few months at a time. As the title suggests, that's not happening. To make make matters more confusing, I have a similar system that works by responding to the values read from the sensors, and turns on the grow light or pump as needed when indicated by the sensors. The only difference in the code is instead of a timer, it responds to the sensor input. It works, and works well. In my last test, it ran for a month and a half, and only stopped because I was using a submersible water pump that ended up corroding (I have a peristaltic pump now).

Ideally, both of these systems are started by opening up a terminal on the pi, and running python3 arduinoDriver.py

Things I've tried

• Restarted the raspberry pi running the timer system
• Updated os via prompt in the top right of the desktop that says "updates are available" or something like that
• Set Try/Except throughout the code, including a global try/catch arounds the arduinoDriver file itself, where in the except statements, used the logging module to write output to a file on the machine. We ran the script, and the next day, upon seeing that it had stopped, checked the log file, only to see that nothing had been logged
• Took the sd card out of the raspberry pi, and put it into a different raspberry pi 3b+
• Re-parented the script by running the command setsid python3 arduinoDriver.py < /dev/zero &> /dev/null & as mentioned in this post
• Running the script through tmux

System Specs

Raspberry pi model 3b+ Raspios Buster 05/07/2021 is the name the iso file has from the download. I can't find it on their website, and as mentioned above, I updated it as a last ditch attempt to figure out what was wrong before coming here. When I get back to the school tomorrow, I can get the specific release if that would be helpful Arduino Mega 2560

Things I know

• When the script runs, the print statements showing the input from the sensors work, so I know the mega and the sensors are working
• When I get to the system after the process crashes, the terminal window that had it open, as well as any other windows that were opened, are closed. I opened up the browser on the pi itself to start troubleshooting, and I was connected to the internet. This is important because every time the raspberry pi restarts, we have to connect it to the school's guest wifi. This means the system didn't turn off and back on
• From the database table storing our sensor data, we can tell how long (approximately) the script was running. It's lasted anywhere from 15 minutes to 4.5 hours
• The empty log file screams that this is a hardware issue. The fact that this is an issue on 2 separate raspberry pis suggests that this is a software issue
• The only issues I'm aware of that sort of fit the symptoms are memory related. Maybe something incremented to a value too large for the data type to hold, causing a memory issue. Maybe the 1gb of ram on the pi is not enough. From the attached code though, I don't think anything is actually ever incrementing, but rather being overwritten. It still doesn't make sense though when you consider the fact that our sensor-responsive system works perfectly.

The 2 systems were meant to run against each other as a comparison for my (and my 2 classmates') senior project. Given the circumstances, our professors (who are also stumped by this problem) have allowed us to axe the timer system from the research, since the plants need time to grow before finals in May. We're now simply comparing the sensor-reactive system to the efforts of humans watering and taking care of a plant by hand. One of the professors suggested that I reach out to this board and ask for help. I'll be checking in on this relatively frequently, so please let me know if I can supply any more information! I realize the code isn't the most beautiful thing, but like I said, the sensor-reactive code is very similar, so we didn't want to change how it functioned so that way we could keep the research comparable

The script

# Third Party
from datetime import datetime, timedelta
import serial
import time
import logging

# Proprietary
from controllers.sendEmail import notifyLowWater, notifyWaterFilled
from controllers.sendData import checkIfDataNeedsSent
from controllers.signalArduino import determineSignalToSend
from controllers.waterPump import checkIfPumpNeeded
from controllers.lightValue import checkIfLightNeeded
from controllers.dataArray import DataArray
from controllers.database import Database


LOG_FORMAT = "%(levelname)s %(asctime)s - %(message)s"

logging.basicConfig(filename="error_log.log",
                    filemode="w",
                    format=LOG_FORMAT,
                    level=logging.ERROR)

#Sets up the mega 2560
board = serial.Serial(
    port = '/dev/ttyACM0',
    baudrate = 115200,
    timeout = None,
)

#Default starting values, needed just to have these guys be global
temp = 70
hum = 20
moisture = 0

timeToKeepLightOn = timedelta(hours=8)
timeToKeepLightOff = timedelta(hours=16)
# timeToKeepLightOn = timedelta(minutes=5) These 2 lines were for testing the script in the classroom
# timeToKeepLightOff = timedelta(seconds=20)They Worked
lightStartOn = datetime.now() - timeToKeepLightOn #When the light turned on
timeLightOn = 0 #This is how long the light has actually been on
isLightOn = False
lightBool = True #This is for the while board.inWaiting() to make the function only execute once
lightOn = False
# lightArray = DataArray(101, 20)

#These are for the same purposes as the light variables above, but for our pump
timeToKeepPumpOn = timedelta(seconds=5)
timeToKeepPumpOff = timedelta(hours=18)
pumpStartTime = datetime.now() - timeToKeepPumpOn
timePumpOn = 0
isPumpOn = False
pumpBool = True

floatFlag = 'LOW' #This is so we know if we have water in our resevoir from our float sensor
emailSent = False
emailTimestamp = 0

moistureHigh = 450 #Not important. We copied our sensor-reactive code into a new file and then added the time delta stuff above to make this for a timer
moistureArray = DataArray(moistureHigh, 5)

timeDataCollected = 0 #Just a time stamp for when we store it. Honestly redundant
lastMinuteSent = 1 #We store our data on the 15, 30, 45, and 0 mark of every hour. This ensures we don't collect duplicates
envId = 1 #We built our schema to be expandable. This is unimportant, just need it to be able to store our data
signalSentBool = False #Checks if we have sent a signal to the mega or not, so we only do it once

db = Database()

def checkIfEmailNeeded(floatFlag, emailTimestamp): #Commented out, not important
    global emailSent
    currentTime = time.time()
    if(currentTime - emailTimestamp > 86400):#86400 seconds in 24 hours
        emailSent = False
    if(floatFlag == 'LOW' and not emailSent):
        notifyLowWater(currentTime)
        emailSent = True
        emailTimestamp = time.time()
    if(floatFlag == 'HIGH' and emailSent):
        notifyWaterFilled(currentTime)
        emailSent = False
    return emailTimestamp

while True:
    try:
        while(board.inWaiting() == 0):
            #emailTimestamp = checkIfEmailNeeded(floatFlag, emailTimestamp)
            if pumpBool:
                pumpStartTime, isPumpOn, endTime = checkIfPumpNeeded(floatFlag, pumpStartTime, isPumpOn, timeToKeepPumpOn, timeToKeepPumpOff) #Returns the time the pump started, if it's currently on or not, and if the pump stopped
                if endTime:
                    timePumpOn += int((datetime.now() - pumpStartTime).total_seconds()) #If it stopped, compute how long it was on
                pumpBool = False
            returned = checkIfDataNeedsSent(lastMinuteSent, temp, hum, moistureArray.getAvg(), lightStartOn, timePumpOn, timeDataCollected, envId, db)
            if returned != lastMinuteSent: #Returned is a 0, 15, 30, or 45, just to make sure we don't send data a bunch of times
                lastMinuteSent = returned
                timeLightOn = 0
                timePumpOn = 0
            if lightBool:
                lightStartOn, isLightOn, endTime = checkIfLightNeeded(lightStartOn, isLightOn, timeToKeepLightOn, timeToKeepLightOff) #Same logic as the pump
                if endTime:
                    timeLightOn += int((datetime.now() - lightStartOn).total_seconds()/60)
                lightBool = False
            if not signalSentBool:
                determineSignalToSend(isPumpOn, isLightOn, board)
                signalSentBool = True
        timeDataCollected = datetime.now().strftime('%Y-%m-%d-%H:%M:%S')
        output = board.readline().decode('utf-8', 'ignore').strip().split(',')
        if len(output) == 6: #reads data from sensors
            # temp = output[0]
            # hum = output[1]
            moisture = int (output[2])
            moistureArray.add(moisture)
            # lightArray.add(output[3])
            if 'LOW' in output[4]:
                floatFlag = 'LOW'
            else:
                floatFlag = 'HIGH'
            pumpBool = True
            lightBool = True
            signalSentBool = False
            logging.info(output)
        else:
            logging.info("Incomplete board output: ", output)
            continue
    except Exception as error:
        logging.error('**Error reading board: ', error)

Answer 1

It's essential to collect error information instead of discarding it. In your first example, the terminal exits for some unknown reason, discarding the terminal output; in your second example, you're explicitly throwing away the output with "&> /dev/null"

Run your command with output/error capturing, using shell redirection:

python3 arduinoDriver.py >> log_file 2>> err_file

This will give you a clue of the failure point, or at least you can add some print() statements to your python code to see what's going on before the shell exits abnormally.

---

Redirections (Bash Reference Manual)

Answer 2

Here's a set of systemd service files that should work for you. The first one is the simplest, for the process that responds to sensor data and thus must always be running.

[Unit]
Description=Start foobar

[Service]
Type=simple
StandardInput=null
StandardOutput=append:/var/log/foobar.log
StandardError=inherit
ExecStart=/path/to/executable

[Install]
WantedBy=default.target       

I'll step through this:

• I have not checked if Description is absolutely required, but since systemd uses this in some output, you probably want one.

• Type=simple is good for programs that do not fork themselves into the background and exit the foreground, something which must happen with almost all init services (systemd is the init implementation most commonly used on contemporary linux systems). There's really only one foreground, only one process can use it at a time, and your should not be it. Beware that if your process does fork itself, you need to use Type=forking. If you aren't sure what forking refers to and you wrote the program being controlled, don't worry (you aren't doing it, stick with simple).

• The standard input/output/error streams of a process run in the foreground are hooked up to some kind of interactive interface such as a command-line terminal. They can be hooked to other things, however, and since we can't (and should not, it can cause problems) use the foreground in a background process, these need to be explicitly dealt with either by the program itself (which can, eg., simply close them) or its parent process, in this case systemd.

  • StandardInput=null means the process will receive no input; based on your description it does not make use of any, so this is appropriate. It's actually the default, so this line is really superfluous.
  • StandardOutput= covers the stdout of your process, ie., what it normally prints to the screen. Hopefully the target ("append:/file/path") is self explanatory; if you want to start a new file each run (erasing the old one), use file instead of append.
  • StandardError=inherit gives it the same value as StandardOutput, which is probably what you want in this case. Uncaught exceptions will be this, as well as anything your program explicitly writes to stderr. If you do that a lot, you may or may not want to use a different value here.

• ExecStart= is the meat and potatoes so to speak. This must be an executable path. If you normally run a script via python3 ... rather than relying on a shebang (#!/usr/bin/env python3) and execute permissions, this should be /usr/bin/env python3 /path/to/your/script.

• Finally, the WantedBy= target is used when the service is installed. Using default.target ensures it runs at boot. If you don't want that, leave this line out. It can still be run via systemctl start ____ if it is in the correct place (see "Installing" below) and will then be considered a "static service".

All this stuff is documented in man pages, of which systemd has quite a few. There is a man systemd.index, and more usefully, a man systemd.directives which lists all the directives (StandardInput, ExecStart, etc) with the specific man page they are documented in. Man pages can be tedious and perhaps convoluted until you get use to their norms and structures, but being able to read them is mandatory if you want to make serious use of a linux system. They are also available online various places, just beware they are often paired to specific versions of the software they document.

Installing a service

1. Copy the service file into /etc/systemd/system (there are other places it can go, but this is the most straightforward and reliable). The filename should be _____.service, where _____ is the name of the service used with, eg., systemctl start ____. You need superuser privileges to do this (sudo cp ...).
2. Run sudo systemctl enable ____.service. You will get a line informing you a symlink has been created. If you don't want the service run at boot and left out a WantedBy=, don't do this. However, to be useful, the service file should still go into /etc/systemd/system.

!Important: If after enabling/installing a service you want to change something in the service file, you need to run sudo systemctl daemon-reload after you have saved the changes.

Timers

The other two things you want to do are timed, and your question implies they simply sleep for (long) specific periods. If instead you want them run on a timer, you need two files, one for the service and one for the timer. The service file will be much the same, but without a WantedBy line.

A basic timer file looks like this:

[Unit]
Description=Timer for foobar

[Timer]
OnBootSec=30
OnUnitActiveSec=64800   
Unit=foobar.service

[Install]
WantedBy=timers.target   

• OnBootSec indicates how long after boot the timer should trigger.
• OnUnitActiveSec indicates how long after the previous run of the "Unit" service the timer should trigger (again); 64800 seconds = 18 hours. Note this only works if there is a "previous run" of the service, not extending beyond a reboot. The OnBootSec should ensure that there is one.
• Unit is the name of the .service file to which this applies.

That combination should run the service at boot and every 18 hours thereafter. If there's some reason you don't want it to run frequently when you are frequently rebooting, you could make OnBootSec longer, or (probably better) include a check in your program.

To install a timer:

• The service does not need to be installed/enabled, but the file does need to be in /etc/systemd/system.
• Place the timer file in the same place with a .timer suffix instead of .service.
• Run sudo systemctl enable ____.timer, then sudo systemctl start ____.timer. It's important to do both. To check that this worked, run systemctl list-timers. You'll get a list, find the name of yours in the UNIT column. The LAST time indicated can be n/a (indicating it hasn't fired yet) but the NEXT time must be real. If it is n/a, or isn't listed at all, you forgot to do at least one of the two commands just mentioned.

Answer 3

A few suggestions which may help.

Don't try to put everything in a single program, break up into discrete tasks.

"turn the grow light on after 16 hours to run for 8 hours" write a program to turn the light on and another to turn off.

"the pump will turn on after 18 hours and run for 5 seconds" write a program to turn the pump for 5 seconds.

"sensors measuring temperature, humidity, ambient light level, soil moisture, and water level" presumably you want to do this regularly (but don't say how long). Again write a program to log data and run every x minutes.

"started by opening up a terminal on the pi" DON'T run in a terminal unless you need to look at the result. Write a program and start with a systemd script which runs in the background and restarts on system failure. This can be done many other ways, but systemd is the best practice, even if it takes a bit of learning.

See https://raspberrypi.stackexchange.com/a/105549/8697 which is a logger I run on my Pi every 15 minutes and shows how simple it is to use cron. It has been running for over 2 years through numerous power failures. This is somewhat dated but I have never felt the need to update it.