I have about 30 Python scripts that I have scheduled to run via cron. Usually, I can run cronjobs on a Raspberry Pi just fine but I usually also only do a handful only.

These scripts are essentially 3 parts:

  1. Pull data from vertica (which can sometimes take up to 30-40 minutes)
  2. Write that to a file
  3. Push that data to an API endpoint

What is odd is I can manually run the 3 methods I tried but whenever I use cronjobs, it was always spotty - where starting from the first Python script, only a handful run through (around 8-12).

I tried 3 methods with cron:

  1. Schedule cronjobs of each of those python scripts with 30 minute gaps
  2. Put the list of python scripts into a shell script to run so that one finishes after the other
  3. Similar to #2 but run tmux session to emulate running it as close to me manually running the shell script

I have even split up the 30 Python scripts into increments of 8 with those methods above but I don't get any errors that I can see. I'd understand if I'm not able to run the script manually but they do work when I do it outside of cron. Any ideas?

This is what I'm seeing through grep -i cron /var/log/syslog:

Dec 31 21:55:01 raspberrypi CRON[3123]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 31 21:55:01 raspberrypi CRON[3124]: (pi) CMD (/home/pi/Documents/run_bs3.sh)
Dec 31 22:00:39 raspberrypi systemd[1]: Started Run anacron jobs.
Dec 31 22:00:39 raspberrypi anacron[3149]: Anacron 2.3 started on 2018-12-31
Dec 31 22:00:39 raspberrypi anacron[3149]: Normal exit (0 jobs run)
Dec 31 22:00:39 raspberrypi systemd[1]: anacron.timer: Adding 6.427877s random time.
  • Please post how exactly you run your scripts manually & via cron. – Dmitry Grigoryev Jan 7 '19 at 9:38

It's possible there's some sort of race condition that occurs only when processes are being run through Cron as you described. Not all errors are logged to syslog when using Cron.

Try changing the mailto address in Cron's settings as described here: Add a MAILTO=my.offsite.email@example.org line at the top of your crontab (replacing the email with your own of course). Then, when your scripts run, you should get an email with the output and errors, which should help you diagnose the problem.

| improve this answer | |
  • Appreciate the response! I tried this and tested running the second chunk of my script that usually is where it doesn't successfully run my python scripts and I'm not seeing any emails - yet only 1 of the pack of python scripts ran. Is there anywhere else I can check for error logs? – macroman Jan 1 '19 at 23:08
  • If you can't see any emails, it seems like Cron's not even attempting to start your script for some reason. Perhaps this is due to a typo in your crontab file? I don't have much experience with Cron, sorry. – Breq Jan 1 '19 at 23:12
  • I'm not sure why the email is not working, but I do know that Cron works. When I test with simple tasks like writing to a file every 5 minutes - everything works as it should. My other Cron tasks are also running properly. It's just this large batch of python scripts that will run properly manually - just not when done via Cron. – macroman Jan 1 '19 at 23:26

I’ve found the issue. I was updating these scripts via Mac and then pushing over to the Raspberry Pi. Somewhere along the way I must have either missed or didn’t successfully push the updated scripts with the exact file paths needed so that manual runs would work because those looked in the same directory for all the references while Cron couldn’t find the files in the same manner. When I read more about Cron logging and a user from another thread mentioned adding to the end of the Cron line:

>>/home/pi/log.txt 2>&1

This helped show the file path issue Cron was having so I went and updated them. Seems to all be working now. Apologies here, looks like it was me, not Cron!

| improve this answer | |
  • Congratulations on solving your problem. Just a couple of items if I may: 1.) Your posted answer is only a fragment of a crontab entry. The answer will be much more useful to others if you include the complete line of your crontab entry. A "FROM - TO" format would be better still. A link to the "user from another thread" would also be appreciated. 2.) You can (and should) upvote your own answer, and eventually (when the system allows it) designate it as "THE" answer (green check mark). As incentive, I'll also upvote your answer if you'll make the changes I suggested under my 'item 1.)'. – Seamus Jan 2 '19 at 10:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.