4

I use my Raspbery PI model B for capturing images for a timelapse. I capture an image every 30 seconds (roughly) using the build in raspistill command within a simple bash script seen below:

#!/bin/bash

imagePath="/home/pi/weatherPi-images/"

echo "Press [CTRL+C] to stop.."
echo "AutoImage process started @ "`date +"%FT%H_%M_%S"`
while :
do
  timestamp=`date +"%FT%H_%M_%S"`
  imageName=$imagePath$timestamp".jpg"
  echo "Taking picture $imageName"
  raspistill -o $imageName -e jpg
  sleep 30
done

Regardless of what I do after sometimes a few hours or this time after four days of straight image capturing (7600+ images) the Pi freezes / the camera LED stays red and the pi is not responding anymore. There is no recurring time interval between these crashes.

The only way to fix this is by cutting the power input from the Pi and restarting it.

After some research most people told that the cause for such a behaviour is a low power- / quality PSU. But in my case the PSU is no problem. It is a 5V/5A/25W power supply which is directly connected to the mains power (which is stable over the year / no blackouts) and has a big power overhead for such a use case. The output voltage is set to ~ 5.1V (±0.05V) so even if the Raspberry Pi draws a lot of power, the voltage is never dropping below 5V.

Right before the Raspberry Pi crashed this time I got this in the : kern.log (click to see more)

It seems that the raspistill script which is written in Python is hanging / filling up all memory.

What can I do to avoid such crashes? If my Pi stays as unreliable for image capturing as it is right now I cannot use it for a long term timelapse (1+ years).

One more thing to mention is that if the outside temperature is higher, the more likely the Pi crashes. The Pi is mounted outside and the temperatures here were lower than 0°C / 32°F for the last Week, today the temperatures were above freezing point again.

The Pi's case temperature has been at around 10°C (50°F)for the past week and because of the higher outside temperatures was going up to 15-18°C (59-64.4°F)today

My Raspberry Pi is overclocked with raspi-config to 1 GHz, this was never an issue as the CPU has a heatsink on it and is always below 40°C (104°F). The Pi had an uptime of 35 days before the crash.

System information:

lsb_release -a

No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 8.0 (jessie)
Release:    8.0
Codename:   jessie

uname -a

Linux horizon 4.1.13+ #826 PREEMPT Fri Nov 13 20:13:22 GMT 2015 armv6l GNU/Linux

lsusb

Bus 001 Device 004: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. LAN9500 Ethernet 10/100 Adapter / SMSC9512/9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

EDIT:

raspistill is written in C. This makes the crash even more strange to me.

EDIT2:

Some more research brought me to this article. Testing the suggested fixes now, for a long timespan (7+ days).

  • Looks like there are 4-5 python processes involved. You don't have any idea what they are? Try ps -A | grep py. – goldilocks Jan 23 '16 at 17:09
  • The only python script that is running is for reading out sensor data and saving it to a remote mysql database. This process uses almost zero IO. – Flatron Jan 23 '16 at 17:16
  • Okay so there is no possibility it is actually consuming all the RAM? Being targeted by the OOM killer does not necessarily mean it was the culprit but it usually does (it also usually prints a table summarizing the evaluation but I don't see it there). If it was hung on IO that would be a hardware failure. You may be interested in cognitivedissonance.ca/cogware/plog – goldilocks Jan 23 '16 at 17:20
  • Thanks this looks interesting. Hmm I saw something about the mysql client beeing force closed in the whole kern.log but I think this was because of something else using up all RAM – Flatron Jan 23 '16 at 17:23
  • @goldilocks maybe this helps, this is additional output of the crash paste.ee/p/8MIp4#OmDMfeyDTqOX34WeA7iU9WQOdhSAQoet until reboot – Flatron Jan 23 '16 at 17:25
4

The OOM killer prints a table summarizing its evaluation when it decides it must kill a process. Generally the biggest one goes. That table is at line 244 in your paste from comments, it begins:

Jan 23 16:16:10 horizon kernel: [3070531.743375] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Jan 23 16:16:10 horizon kernel: [3070531.743405] [   96]     0    96     2954        1       7       0      167         -1000 systemd-udevd
Jan 23 16:16:10 horizon kernel: [3070531.743421] [  467]     0   467     1259       39       6       0       18             0 cron
Jan 23 16:16:10 horizon kernel: [3070531.743437] [  471]     0   471      956       31       4       0       33             0 systemd-logind
Jan 23 16:16:10 horizon kernel: [3070531.743454] [  480]   106   480      987       57       6       0       19             0 avahi-daemon
Jan 23 16:16:10 horizon kernel: [3070531.743470] [  483]   105   483     1390       57       5       0       52          -900 dbus-daemon
Jan 23 16:16:10 horizon kernel: [3070531.743485] [  493]     0   493      634      120       5       0       37             0 dhcpcd
Jan 23 16:16:10 horizon kernel: [3070531.743501] [  496]   106   496      961       15       6       0       40             0 avahi-daemon
Jan 23 16:16:10 horizon kernel: [3070531.743516] [  507] 65534   507      565        4       5       0       26             0 thd
Jan 23 16:16:10 horizon kernel: [3070531.743531] [  528]     0   528     7838      211      11       0       85             0 rsyslogd
Jan 23 16:16:10 horizon kernel: [3070531.743546] [  626]   107   626     1413       48       6       0       64             0 ntpd
Jan 23 16:16:10 horizon kernel: [3070531.743561] [  666]     0   666     1779       29       6       0       72             0 wpa_supplicant
Jan 23 16:16:10 horizon kernel: [3070531.743577] [  672]     0   672     1005        1       6       0       30             0 agetty
Jan 23 16:16:10 horizon kernel: [3070531.743592] [  673]     0   673      960        1       5       0       30             0 agetty
Jan 23 16:16:10 horizon kernel: [3070531.743607] [  720]     0   720    25347     1338      22       0      690             0 php5-fpm
Jan 23 16:16:10 horizon kernel: [3070531.743622] [  739]     0   739      471        1       5       0       26             0 mysqld_safe
Jan 23 16:16:10 horizon kernel: [3070531.743637] [  800]    33   800    25347        9      20       0      707             0 php5-fpm
Jan 23 16:16:10 horizon kernel: [3070531.743652] [  801]    33   801    25347        9      20       0      707             0 php5-fpm
Jan 23 16:16:10 horizon kernel: [3070531.743667] [ 1243]   110  1243    79494      461      41       0     6867             0 mysqld

There's more, including at least 50 python processes that seem to have been spawned in conjunction with cron and sh processes, which are also still running.

The last one above is mysqld, which has a modest amount of RSS (461 KB) but by far the largest virtual mem size (~80 MB). Of course, the real culprit is the hundreds of cron/sh/python instances.

Here's a guess: After mysqld is gone, since it was not the real cause of the problem, the problem persists, and the kernel decides to off one of the python processes. However, it is (possibly, I'm guessing) in an uninterruptible sleep state ('D' in tools such as top). This is caused by a hardware I/O failure. For technical reasons, it is not something the kernel will do anything about except wait; if it is a device containing the root filesystem then there may be piles of processes suffering the same fate and the system is fubar'd at that point.

The fact that it was able to print a log in that case is interesting, but I/O failure may be because of only a specific part (and not all) of a storage device. Hence, some things accessing it will be fine, others will be stuck in D. The normal approach with a spinning disk is to then run a bad blocks check to isolate them and decide whether to chunk it, or just blacklist the bad blocks. With an SD card, the latter is not an option.

  • Thank you very much for your detailed answer. I currently have not that much time but I'll look into this. I will update my question upon my findings in the next few days. – Flatron Jan 24 '16 at 9:54
1

I had the same problem. My Pi Zero W worked fine for 10 hours or so and then various kernel panic messages. The errors would get more frequent and at a point I could no longer connect to the Pi.

It looks like the Pi is under-volted for this job.

After some investigation I solved this by adding the following lines to the file /boot/config.txt

over_voltage=4
force_turbo=1

The problem hasn't appeared, the Pi is running for almost 24 hours without interruption.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.