1

I've set up a weekly backup cron job using tar and xz. Lately I'm getting the xz process killed by the kernel because of lack of memory. From /var/log/errors.log:

Jun 30 03:52:09 pi kernel: [1155695.358531] Out of memory: Kill process 31379 (xz) score 148 or sacrifice child
Jun 30 03:52:09 pi kernel: [1155695.398374] Killed process 31379 (xz) total-vm:97384kB, anon-rss:84296kB, file-rss:96kB
Jun 30 11:28:01 pi kernel: [1183048.438964] Out of memory: Kill process 12829 (xz) score 124 or sacrifice child
Jun 30 11:28:01 pi kernel: [1183048.465286] Killed process 12829 (xz) total-vm:100656kB, anon-rss:72860kB, file-rss:60kB

Here's an excerpt of what free -m -s 5 reports in 5 second intervals after starting the backup script just before and after it gets killed. The last output is after it's killed:

             total       used       free     shared    buffers     cached
Mem:           461        280        181          0          0         65
-/+ buffers/cache:        215        246
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           461        284        176          0          0         66
-/+ buffers/cache:        217        243
Swap:            0          0          0
             total       used       free     shared    buffers     cached
Mem:           461        286        175          0          0         65
-/+ buffers/cache:        221        240
Swap:            0          0          0

             total       used       free     shared    buffers     cached
Mem:           461        209        252          0          0         70
-/+ buffers/cache:        137        323
Swap:            0          0          0

It seems that there's around 175 MB free memory. Judging by the memory usage variation, which appears to be a few MB at each interval, it seems to me to be unlikely that suddenly all remaining free memory would get consumed within a 5 second interval.

Here's output from vmstat 5 from another run (also 5 second interval):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 236156   3512  92484    0    0     0     0  349  190 24  9 67  0
 0  0      0 236156   3516  92532    0    0    11     0  353  264 15 12 73  0
 1  0      0 216344   3548  93496    0    0   195     7  477  287 28 10 61  0
 1  0      0 215700   2424  85760    0    0   230    10  482  331 91  9  0  0
 1  1      0 214368    688  80700    0    0   217     2  476  285 94  6  0  0
 2  1      0 208120     72  78092    0    0   260     3  494  246 94  6  0  0
 3  1      0 206816     60  73072    0    0   250     2  495  300 93  7  0  0
 1  0      0 206648     60  68584    0    0  2158     2 1309  438 88 12  0  0
 1  0      0 201720     60  69044    0    0  2278     2 1310  525 85 15  0  0
 5  0      0 197880     60  67124    0    0  4621   470 2255  624 82 16  0  2
 1  1      0 192868     60  70148    0    0 13839     6 5145 1159 59 29  0 11
 1  1      0 193496     60  65964    0    0 12668     3 5124 1199 61 31  0  8
 2  3      0 189672     60  66836    0    0 15159     3 6232 1475 54 36  0 10
 2  0      0 185560     60  68920    0    0 14072     3 5729 1301 64 30  0  6
 0  8      0 184084     60  65884    0    0 10928   123 4081  945 71 23  0  6
 0  0      0 254680    488  75192    0    0  9922    39 4391 1275 55 30  7  8

On this run free memory seems to go down to about 180 MB before the process gets killed.

Any ideas what could be causing this or how to troubleshoot?

How should an error.log line like Jun 30 11:28:01 pi kernel: [1183048.465286] Killed process 12829 (xz) total-vm:100656kB, anon-rss:72860kB, file-rss:60kB be interpreted with regard to memory?

update: Here's what ulimit -a reports:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1805
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1805
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
1
  • what's the point in backing up entire file system? checking if wear-leveling works for your SD card? please, leave alone the system stuff, and backup only your files that you really need. – lenik Jun 30 '13 at 15:28
2

Your total virtual memory seems large for the RPi, 100,565 kB = 98.2 MB, which sounds like there might be a 100 MB cap.

Try running ulimit -a, if there is a limit on the virtual memory you could raise it using ulimit -v 204800 (200 MB) and see if that gets around the problem.

See here for more details

Update

Since it does not appear to be a system limit, I can only imagine that something (maybe xz, maybe something else) is trying to reserve a large amount of memory briefly, and therefore tipping things over the edge.

Could you try adding a swap file and see what happens:

sudo su
dd if=/dev/zero of=/swapfile bs=1024 count=131072
mkswap /swapfile
swapon /swapfile
echo "/swapfile swap swap defaults 0 0" >> /etc/fstab

This should add a 128 MB swap file, make sure to exclude it from the backup (change /swapfile to be somewhere else if need be, e.g. /backup/swapfile).

6
  • Thanks for suggesting it. It looks like virtual memory is unlimited. I've updated my question with its output. – seron Jun 30 '13 at 13:16
  • @seron hmm, not too sure what is going on here then, may I ask what you are backing up? I do not have an SD card with archlinux to hand, but with raspbian I have 99 MB of swap listed, I wonder if adding a swap file would help (something may be reserving memory ahead of time)? – PiBorg Jun 30 '13 at 14:02
  • It backups the entire file system with a few exceptions (the backup directory being one of them). When it completes, as it's done many times before and still occasionally does, it occupies about 250 MB of storage. – seron Jun 30 '13 at 14:08
  • @seron Added a swap suggestion. – PiBorg Jun 30 '13 at 14:23
  • 1
    Thanks for your suggestion. I rebooted the Pi (without adding swap) and the problem hasn't manifested itself again. It completed an on demand backup after reboot without problems and has completed a weekly cron controlled backup too without problems. I don't know if the underlying problem is gone but the symptoms are absent so far. – seron Jul 13 '13 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.