I have about 30 Python scripts that I have scheduled to run via cron. Usually, I can run cronjobs on a Raspberry Pi just fine but I usually also only do a handful only.
These scripts are essentially 3 parts:
- Pull data from vertica (which can sometimes take up to 30-40 minutes)
- Write that to a file
- Push that data to an API endpoint
What is odd is I can manually run the 3 methods I tried but whenever I use cronjobs, it was always spotty - where starting from the first Python script, only a handful run through (around 8-12).
I tried 3 methods with cron:
- Schedule cronjobs of each of those python scripts with 30 minute gaps
- Put the list of python scripts into a shell script to run so that one finishes after the other
- Similar to #2 but run tmux session to emulate running it as close to me manually running the shell script
I have even split up the 30 Python scripts into increments of 8 with those methods above but I don't get any errors that I can see. I'd understand if I'm not able to run the script manually but they do work when I do it outside of cron. Any ideas?
This is what I'm seeing through
grep -i cron /var/log/syslog:
Dec 31 21:55:01 raspberrypi CRON: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Dec 31 21:55:01 raspberrypi CRON: (pi) CMD (/home/pi/Documents/run_bs3.sh) Dec 31 22:00:39 raspberrypi systemd: Started Run anacron jobs. Dec 31 22:00:39 raspberrypi anacron: Anacron 2.3 started on 2018-12-31 Dec 31 22:00:39 raspberrypi anacron: Normal exit (0 jobs run) Dec 31 22:00:39 raspberrypi systemd: anacron.timer: Adding 6.427877s random time.