1

My raspberry pi 4 is set to run a python script every hour that turns on some philips hue lights and takes a picture with the camera module. It worked perfectly for 26 days, then it stopped running. If I reboot it, it works again for a few hours, then stops working.

When in the broken state, the syslog shows the cron job trying to run my script, but it neither turns on the lights nor takes a picture. When in this state, I also can no longer use a VNC client to access the raspberry pi, but can SSH in.

Running top, the memory and cpu don't look particularly taxed, but I don't really know what to look for.

In the syslog, you can see my script run at 3:25, then there is a kernel problem right after. The raspberry pi did take the picture at 3:25, but then failed after that until I rebooted:

Apr  2 03:17:01 raspberrypi CRON[9893]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Apr  2 03:25:01 raspberrypi CRON[11726]: (pi) CMD (python3 /home/pi/cameratest/timecamera-cron.py)
Apr  2 03:27:51 raspberrypi kernel: [28383.750583] INFO: task kworker/1:2:17982 blocked for more than 120 seconds.
Apr  2 03:27:51 raspberrypi kernel: [28383.750594]       Tainted: G         C        4.19.97-v7l+ #1294
Apr  2 03:27:51 raspberrypi kernel: [28383.750602] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  2 03:27:51 raspberrypi kernel: [28383.750611] kworker/1:2     D    0 17982      2 0x00000000
Apr  2 03:27:51 raspberrypi kernel: [28383.750640] Workqueue: events dbs_work_handler
Apr  2 03:27:51 raspberrypi kernel: [28383.750669] [<c09c8f5c>] (__schedule) from [<c09c95cc>] (schedule+0x50/0xa8)
Apr  2 03:27:51 raspberrypi kernel: [28383.750686] [<c09c95cc>] (schedule) from [<c09cd5b4>] (schedule_timeout+0x200/0x428)
Apr  2 03:27:51 raspberrypi kernel: [28383.750700] [<c09cd5b4>] (schedule_timeout) from [<c09ca23c>] (wait_for_common+0xd4/0x1b0)
Apr  2 03:27:51 raspberrypi kernel: [28383.750713] [<c09ca23c>] (wait_for_common) from [<c09ca338>] (wait_for_completion+0x20/0x24)
Apr  2 03:27:51 raspberrypi kernel: [28383.750728] [<c09ca338>] (wait_for_completion) from [<c0840b14>] (rpi_firmware_transaction+0x78/0xd0)
Apr  2 03:27:51 raspberrypi kernel: [28383.750745] [<c0840b14>] (rpi_firmware_transaction) from [<c0840cac>] (rpi_firmware_property_list+0x140/0x2b0)
Apr  2 03:27:51 raspberrypi kernel: [28383.750760] [<c0840cac>] (rpi_firmware_property_list) from [<c0840e98>] (rpi_firmware_property+0x7c/0xfc)
Apr  2 03:27:51 raspberrypi kernel: [28383.750776] [<c0840e98>] (rpi_firmware_property) from [<c06ae5d4>] (raspberrypi_clock_property+0x4c/0x74)
Apr  2 03:27:51 raspberrypi kernel: [28383.750792] [<c06ae5d4>] (raspberrypi_clock_property) from [<c06ae640>] (raspberrypi_fw_set_rate+0x44/0x98)
Apr  2 03:27:51 raspberrypi kernel: [28383.750805] [<c06ae640>] (raspberrypi_fw_set_rate) from [<c06ae70c>] (raspberrypi_fw_pll_div_set_rate+0x38/0x40)
Apr  2 03:27:51 raspberrypi kernel: [28383.750820] [<c06ae70c>] (raspberrypi_fw_pll_div_set_rate) from [<c06a74ec>] (clk_change_rate+0x124/0x474)
Apr  2 03:27:51 raspberrypi kernel: [28383.750835] [<c06a74ec>] (clk_change_rate) from [<c06a79d0>] (clk_core_set_rate_nolock+0x194/0x1a8)
Apr  2 03:27:51 raspberrypi kernel: [28383.750849] [<c06a79d0>] (clk_core_set_rate_nolock) from [<c06a7a20>] (clk_set_rate+0x3c/0x78)
Apr  2 03:27:51 raspberrypi kernel: [28383.750863] [<c06a7a20>] (clk_set_rate) from [<c080dd20>] (dev_pm_opp_set_rate+0x284/0x54c)
Apr  2 03:27:51 raspberrypi kernel: [28383.750878] [<c080dd20>] (dev_pm_opp_set_rate) from [<c08179e4>] (set_target+0x40/0x60)
Apr  2 03:27:51 raspberrypi kernel: [28383.750894] [<c08179e4>] (set_target) from [<c081252c>] (__cpufreq_driver_target+0x2d8/0x5a8)
Apr  2 03:27:51 raspberrypi kernel: [28383.750909] [<c081252c>] (__cpufreq_driver_target) from [<c08160a8>] (od_dbs_update+0xec/0x174)
Apr  2 03:27:51 raspberrypi kernel: [28383.750924] [<c08160a8>] (od_dbs_update) from [<c0816cc4>] (dbs_work_handler+0x3c/0x68)
Apr  2 03:27:51 raspberrypi kernel: [28383.750942] [<c0816cc4>] (dbs_work_handler) from [<c023e028>] (process_one_work+0x170/0x458)
Apr  2 03:27:51 raspberrypi kernel: [28383.750957] [<c023e028>] (process_one_work) from [<c023e36c>] (worker_thread+0x5c/0x5a4)
Apr  2 03:27:51 raspberrypi kernel: [28383.750970] [<c023e36c>] (worker_thread) from [<c02446a0>] (kthread+0x138/0x168)
Apr  2 03:27:51 raspberrypi kernel: [28383.750984] [<c02446a0>] (kthread) from [<c02010ac>] (ret_from_fork+0x14/0x28)
Apr  2 03:27:51 raspberrypi kernel: [28383.750993] Exception stack(0xdc659fb0 to 0xdc659ff8)
Apr  2 03:27:51 raspberrypi kernel: [28383.751003] 9fa0:                                     00000000 00000000 00000000 00000000
Apr  2 03:27:51 raspberrypi kernel: [28383.751014] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Apr  2 03:27:51 raspberrypi kernel: [28383.751024] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Apr  2 03:27:51 raspberrypi kernel: [28383.751037] INFO: task kworker/2:0:11071 blocked for more than 120 seconds.
Apr  2 03:27:51 raspberrypi kernel: [28383.751045]       Tainted: G         C        4.19.97-v7l+ #1294
Apr  2 03:27:51 raspberrypi kernel: [28383.751053] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

there are also errors like this:

Apr  1 19:07:56 raspberrypi dhcpcd[399]: script_runreason: control_queue: No buffer space available
Apr  1 19:07:59 raspberrypi dhcpcd[399]: script_runreason: control_queue: No buffer space available

Any help is appreciated!

  • 3
    Could it have run out of storage (log files filled it up?). Try running 'df' on it to see usage. You could take the output of df (disk usage) and free (memory usage) and save/publish them along with the photo to see what is filling up – Will Apr 7 at 7:46
  • 2
    Something is broken - I'll hazard a guess it's your code. You need a plan to find where the code is broken. You need to isolate the problem. That means making changes to your code, and noting what difference it makes. There's always the possibility that someone will recognize the error message, but these things rarely pinpont the problem. – Seamus Apr 7 at 9:39
  • @Will, when I ran top, it didn't look like either storage or memory were maxed out. I'll try again with df and report back. – John Apr 8 at 0:10
  • @Seamus, I'll try breaking my code into parts and running each individually. When I was first trying to do this, I had the same code but for the iteration every hour, I just had a for loop and sleep in my python code: it stopped working pretty quickly. Do you think that was generating the same problem on a shorter timescale? – John Apr 8 at 0:15
  • I don't have a plan for you @John - you'll just need to start at the bottom, then keep adding things until it breaks. Have you tried running this on another Pi? Do you have a good power supply? – Seamus Apr 8 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.