4

I use my Raspbery PI model B for capturing images for a timelapse. I capture an image every 30 seconds (roughly) using the build in raspistill command within a simple bash script seen below:

#!/bin/bash

imagePath="/home/pi/weatherPi-images/"

echo "Press [CTRL+C] to stop.."
echo "AutoImage process started @ "`date +"%FT%H_%M_%S"`
while :
do
  timestamp=`date +"%FT%H_%M_%S"`
  imageName=$imagePath$timestamp".jpg"
  echo "Taking picture $imageName"
  raspistill -o $imageName -e jpg
  sleep 30
done

Regardless of what I do after sometimes a few hours or this time after four days of straight image capturing (7600+ images) the Pi freezes / the camera LED stays red and the pi is not responding anymore. There is no recurring time interval between these crashes.

The only way to fix this is by cutting the power input from the Pi and restarting it.

After some research most people told that the cause for such a behaviour is a low power- / quality PSU. But in my case the PSU is no problem. It is a 5V/5A/25W power supply which is directly connected to the mains power (which is stable over the year / no blackouts) and has a big power overhead for such a use case. The output voltage is set to ~ 5.1V (±0.05V) so even if the Raspberry Pi draws a lot of power, the voltage is never dropping below 5V.

Right before the Raspberry Pi crashed this time I got this in the : kern.log (click to see more)

It seems that the raspistill script which is written in Python is hanging / filling up all memory.

What can I do to avoid such crashes? If my Pi stays as unreliable for image capturing as it is right now I cannot use it for a long term timelapse (1+ years).

One more thing to mention is that if the outside temperature is higher, the more likely the Pi crashes. The Pi is mounted outside and the temperatures here were lower than 0°C / 32°F for the last Week, today the temperatures were above freezing point again.

The Pi's case temperature has been at around 10°C (50°F)for the past week and because of the higher outside temperatures was going up to 15-18°C (59-64.4°F)today

My Raspberry Pi is overclocked with raspi-config to 1 GHz, this was never an issue as the CPU has a heatsink on it and is always below 40°C (104°F). The Pi had an uptime of 35 days before the crash.

System information:

lsb_release -a

No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 8.0 (jessie)
Release:    8.0
Codename:   jessie

uname -a

Linux horizon 4.1.13+ #826 PREEMPT Fri Nov 13 20:13:22 GMT 2015 armv6l GNU/Linux

lsusb

Bus 001 Device 004: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. LAN9500 Ethernet 10/100 Adapter / SMSC9512/9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

EDIT:

raspistill is written in C. This makes the crash even more strange to me.

EDIT2:

Some more research brought me to this article. Testing the suggested fixes now, for a long timespan (7+ days).

6
  • Looks like there are 4-5 python processes involved. You don't have any idea what they are? Try ps -A | grep py.
    – goldilocks
    Commented Jan 23, 2016 at 17:09
  • The only python script that is running is for reading out sensor data and saving it to a remote mysql database. This process uses almost zero IO.
    – Bent
    Commented Jan 23, 2016 at 17:16
  • Okay so there is no possibility it is actually consuming all the RAM? Being targeted by the OOM killer does not necessarily mean it was the culprit but it usually does (it also usually prints a table summarizing the evaluation but I don't see it there). If it was hung on IO that would be a hardware failure. You may be interested in cognitivedissonance.ca/cogware/plog
    – goldilocks
    Commented Jan 23, 2016 at 17:20
  • Thanks this looks interesting. Hmm I saw something about the mysql client beeing force closed in the whole kern.log but I think this was because of something else using up all RAM
    – Bent
    Commented Jan 23, 2016 at 17:23
  • @goldilocks maybe this helps, this is additional output of the crash paste.ee/p/8MIp4#OmDMfeyDTqOX34WeA7iU9WQOdhSAQoet until reboot
    – Bent
    Commented Jan 23, 2016 at 17:25

5 Answers 5

4

The OOM killer prints a table summarizing its evaluation when it decides it must kill a process. Generally the biggest one goes. That table is at line 244 in your paste from comments, it begins:

Jan 23 16:16:10 horizon kernel: [3070531.743375] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 23 16:16:10 horizon kernel: [3070531.743405] [   96]     0    96     2954        1       7       0      167         -1000 systemd-udevd
Jan 23 16:16:10 horizon kernel: [3070531.743421] [  467]     0   467     1259       39       6       0       18             0 cron
Jan 23 16:16:10 horizon kernel: [3070531.743437] [  471]     0   471      956       31       4       0       33             0 systemd-logind
Jan 23 16:16:10 horizon kernel: [3070531.743454] [  480]   106   480      987       57       6       0       19             0 avahi-daemon
Jan 23 16:16:10 horizon kernel: [3070531.743470] [  483]   105   483     1390       57       5       0       52          -900 dbus-daemon
Jan 23 16:16:10 horizon kernel: [3070531.743485] [  493]     0   493      634      120       5       0       37             0 dhcpcd
Jan 23 16:16:10 horizon kernel: [3070531.743501] [  496]   106   496      961       15       6       0       40             0 avahi-daemon
Jan 23 16:16:10 horizon kernel: [3070531.743516] [  507] 65534   507      565        4       5       0       26             0 thd
Jan 23 16:16:10 horizon kernel: [3070531.743531] [  528]     0   528     7838      211      11       0       85             0 rsyslogd
Jan 23 16:16:10 horizon kernel: [3070531.743546] [  626]   107   626     1413       48       6       0       64             0 ntpd
Jan 23 16:16:10 horizon kernel: [3070531.743561] [  666]     0   666     1779       29       6       0       72             0 wpa_supplicant
Jan 23 16:16:10 horizon kernel: [3070531.743577] [  672]     0   672     1005        1       6       0       30             0 agetty
Jan 23 16:16:10 horizon kernel: [3070531.743592] [  673]     0   673      960        1       5       0       30             0 agetty
Jan 23 16:16:10 horizon kernel: [3070531.743607] [  720]     0   720    25347     1338      22       0      690             0 php5-fpm
Jan 23 16:16:10 horizon kernel: [3070531.743622] [  739]     0   739      471        1       5       0       26             0 mysqld_safe
Jan 23 16:16:10 horizon kernel: [3070531.743637] [  800]    33   800    25347        9      20       0      707             0 php5-fpm
Jan 23 16:16:10 horizon kernel: [3070531.743652] [  801]    33   801    25347        9      20       0      707             0 php5-fpm
Jan 23 16:16:10 horizon kernel: [3070531.743667] [ 1243]   110  1243    79494      461      41       0     6867             0 mysqld

There's more, including at least 50 python processes that seem to have been spawned in conjunction with cron and sh processes, which are also still running.

The last one above is mysqld, which has a modest amount of RSS (461 KB) but by far the largest virtual mem size (~80 MB). Of course, the real culprit is the hundreds of cron/sh/python instances.

Here's a guess: After mysqld is gone, since it was not the real cause of the problem, the problem persists, and the kernel decides to off one of the python processes. However, it is (possibly, I'm guessing) in an uninterruptible sleep state ('D' in tools such as top). This is caused by a hardware I/O failure. For technical reasons, it is not something the kernel will do anything about except wait; if it is a device containing the root filesystem then there may be piles of processes suffering the same fate and the system is fubar'd at that point.

The fact that it was able to print a log in that case is interesting, but I/O failure may be because of only a specific part (and not all) of a storage device. Hence, some things accessing it will be fine, others will be stuck in D. The normal approach with a spinning disk is to then run a bad blocks check to isolate them and decide whether to chunk it, or just blacklist the bad blocks. With an SD card, the latter is not an option.

1
  • Thank you very much for your detailed answer. I currently have not that much time but I'll look into this. I will update my question upon my findings in the next few days.
    – Bent
    Commented Jan 24, 2016 at 9:54
2
over_voltage=4
force_turbo=1

...in /boot/config.txt solved the problem for me with a Pi Zero, good quality power supply and low quality 3rd party knock-off cheap camera.

My guess is that 3rd party cheapo cameras draw more volts that the board expects, and that over volting the board compensates for this.

1

I had the same problem. My Pi Zero W worked fine for 10 hours or so and then various kernel panic messages. The errors would get more frequent and at a point I could no longer connect to the Pi.

It looks like the Pi is under-volted for this job.

After some investigation I solved this by adding the following lines to the file /boot/config.txt

over_voltage=4
force_turbo=1

The problem hasn't appeared, the Pi is running for almost 24 hours without interruption.

1

A 47000uF capacitor across the 5V GPIO pins fixed it for me. Started with 100uF with no luck, didn't have anything in between handy so went with what I had, been running for weeks now with no problem compared to a day or two at best before. Pi zero running RPI cam control FWIW.

2
  • Just pay attention to your power supply: some of them are not happy about over-current on every startup. It's less of a problem if you rarely switch the Pi off though. Commented Feb 15, 2021 at 11:47
  • Worked for me. After extending the 5v power lead several metres it was failing to take a picture and hanging randomly 50% of the time. A 4700uF cap on the lead right next to the pi solved it. Making the lead shorter would have also been a solution but I had constraints I needed to work with.
    – F Chopin
    Commented Mar 6, 2021 at 17:35
0

Same issue with a Pi Zero while trying to set up LED lights using Hyperion. Had an issue where the Pi would shut off and my capture card I was using would also shut off and fail to be recognized. I ran a test and my CPU was just getting hammered from something that shouldn't take too much of it. Entered the code above which has worked great so far.

over_voltage=4
force_turbo=1

I suppose this is a voltage issue given the solution but my CPU is also running normally again. It's also worth also noting that the raspi-config file that you can easily access through SSH didn't work for my Pi Zero as it mentioned you need a Pie 1 or higher to access overclocking settings so had to pull the SD card to my computer and change the file that way.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.