Nginx “Out Of memory: Kill process”

Question

I've been hitting a number off issues with a personal site I've been trying to run off of a raspberry pi 2.

The error logs in nginx have the follow error repeated several times:

[alert] 13551#0: worker process 1709 exited on signal 9

Looking online I eventually found some reference to memory being related to signal 9 issues and the advice was to check syslogs.

Looking through syslog there are quite a few errors along the lines of:

nginx invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
nginx cpuset=/ mems_allowed=0
CPU: 2 PID: 15840 Comm: nginx Not tainted 4.1.13-v7+ #826
Hardware name: BCM2709
[<80018444>] (unwind_backtrace) from [<80013e08>] (show_stack+0x20/0x24)
[<80013e08>] (show_stack) from [<8055a188>] (dump_stack+0x98/0xe0)
[<8055a188>] (dump_stack) from [<80557b00>] (dump_header+0xb0/0x1e4)
[<80557b00>] (dump_header) from [<800f63c0>] (oom_kill_process+0x200/0x3f0)
[<800f63c0>] (oom_kill_process) from [<800f68a4>] (__out_of_memory+0x274/0x380)
[<800f68a4>] (__out_of_memory) from [<800f6b48>] (out_of_memory+0x60/0x80)
[<800f6b48>] (out_of_memory) from [<800fb43c>] (__alloc_pages_nodemask+0x890/0x8ec)
[<800fb43c>] (__alloc_pages_nodemask) from [<8011e370>] (handle_mm_fault+0xbb8/0xf90)
[<8011e370>] (handle_mm_fault) from [<80560964>] (do_page_fault+0x2dc/0x3f4)
[<80560964>] (do_page_fault) from [<800091e4>] (do_DataAbort+0x44/0xc4)
[<800091e4>] (do_DataAbort) from [<805601e4>] (__dabt_usr+0x44/0x60)
Exception stack(0xb4159fb0 to 0xb4159ff8)
9fa0:                                     00001031 00000000 0000daf1 369304e0
9fc0: 369304e8 769864d4 00000063 36931510 76986504 00001028 000001ff 00000001
9fe0: 00001031 7eb148d0 768bea4c 768bf020 60000010 ffffffffMem-Info:active_anon:123769 inactive_anon:87871 isolated_anon:32
active_file:142 inactive_file:265 isolated_file:0
unevictable:0 dirty:0 writeback:2767 unstable:0
slab_reclaimable:1447 slab_unreclaimable:1952
mapped:338 shmem:128 pagetables:800 bounce:0
free:2034 free_pcp:128 free_cma:0Normal free:8136kB min:8192kB low:10240kB high:12288kB active_anon:495076kB inactive_anon:351484kB active_file:568kB inactive_file:1060kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:901120kB managed:883148kB mlocked:0kB dirty:0kB writeback:11068kB mapped:1352kB shmem:512kB slab_reclaimable:5788kB slab_unreclaimable:7808kB kernel_stack:944kB pagetables:3200kB unstable:0kB bounce:0kB free_pcp:512kB local_pcp:240kB free_cma:0kB writeback_tmp:0kB pages_scanned:15608 all_unreclaimable? yeslowmem_reserve[]: 0 0Normal: 2*4kB (UR) 2*8kB (UM) 1*16kB (R) 1*32kB (U) 0*64kB 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 1*4096kB (R) = 8136kB3339 total pagecache pages2776 pages in swap cacheSwap cache stats: add 478548, delete 475772, find 56890/94212Free swap  = 0kBTotal swap = 102396kB225280 pages RAM0 pages HighMem/MovableOnly2445 pages reserved
2048 pages cma reserved
[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[  108]     0   108     2468      153      10       0       59             0 systemd-journal
[  110]     0   110     2960        1       7       0      181         -1000 systemd-udevd
[  345]     0   345     1255       11       6       0       54             0 cron
[  350]     0   350      958       24       6       0       39             0 systemd-logind
[  353]   106   353      964       32       6       0       43             0 avahi-daemon
[  355]   105   355     1368        0       7       0       90          -900 dbus-daemon
[  375] 65534   375      568        0       5       0       35             0 thd
[  381]     0   381      637       32       6       0       61             0 dhcpcd
[  388]   106   388      964        0       5       0       56             0 avahi-daemon
[  441]     0   441     8031       21      11       0      183             0 rsyslogd
[  450]     0   450     1952       29       8       0       97         -1000 sshd
[  497]   107   497     1416       38       7       0       94             0 ntpd
[  623]     0   623     1008        1       7       0       32             0 agetty
[14333]     0 14333     2909       16      11       0      169             0 sshd
[14341]  1000 14341     1231       15       7       0       97             0 systemd
[14346]  1000 14346     1667       10       7       0      297             0 (sd-pam)
[14350]  1000 14350     2942        0       9       0      181             0 sshd
[14353]  1000 14353     1577       15       8       0      369             0 bash
[14399]     0 14399     2909       16       9       0      167             0 sshd
[14409]  1000 14409     2946       14       8       0      195             0 sshd
[14412]  1000 14412      620       15       6       0       82             0 sftp-server
[15709]  1000 15709    22122       15      41       0     3479             0 nodejs
[15839]     0 15839    16685        4      12       0      234             0 nginx
[15840]    33 15840   233000   208490     434       0     7960             0 nginx
[15841]    33 15841    16722        0      12       0      275             0 nginx
[15867]  1000 15867    22167        0      42       0     3790             0 nodejs
[15876]  1000 15876    22121        0      44       0     3501             0 nodejs
[15887]  1000 15887    30995       15      45       0     2837             0 nodejs
[15927]  1000 15927     1273       25       8       0       65             0 top
Out of memory: Kill process 15840 (nginx) score 880 or sacrifice child
Killed process 15840 (nginx) total-vm:932000kB, anon-rss:833880kB, file-rss:80kB

Nginx config is fine if I use it locally (which skips SSL negotiation). Accessing via a url with SSL seems to result in sketchy performance and this looks like it could be the cause.

Has anyone else hit this issue or know of some tips / tricks / advice on checking memory footprint or ways to adapt my config to be a bit tighter in terms of memory usage?

Cheers, Matt

Answer 1

You're obviously running out of memory. Without knowing what exactly you're running on your system it's hard to advise, but check out two main suggestions:

1. Configure your web server. Go though /etc/nginx/nginx.conf and reduce settings which could use RAM (number of simultaneous connections, buffer sizes, etc.) If you're using web scripts (PHP and friends), check out their configs as well.

These settings may be a good start:

worker_processes 1;
http {
    client_max_body_size 10M;
    ...
}

In case of PHP specifically, set memory_limit to 16M in /etc/php5/fpm/php.ini.

2. If that doesn't help, use swap. It will slow your web server further, but at least it should prevent OOM crashes:

dd if=/dev/zero of=/swapfile bs=1M count=1024
mkswap /swapfile
swapon /swapfile

If swap helps, you can activate it automatically at boot time by editing /etc/dphys-swapfile, then running

sudo dphys-swapfile swapoff
sudo dphys-swapfile setup
sudo dphys-swapfile swapon

Answer 2

You are running out of memory and being hit by the OOM Killer in an attempt to stop your system crashing.

The OOM Killer is a bit random, but is weighted by the amount of memory a process uses so if you see a lot of nginx worker processes being killed then they are likely responsible for the memory usage.

You can use tools like free, top or htop (slightly nicer version of top) to see how much memory you are using and which applications are using most of it to help give you an idea of where the problems might be and if the changes you make are actually helping.

Given that it is the nginx worker processes that are being killed how many do you have? I think by default it is the number of cores you have but it can be configured. Since the workers have their own memory pool the more works you have the more memory you will use. If you are using php-fpm you might want to look at the number of worker that uses as well as they share the same problem as the nginx workers.

Raspberry pi images also don't come with swap space (but can be configured to have some) which can give you more effective memory before the OOM Killer kicks in. Swap space would be most useful if you occasionally spike in memory usage or have a lot of generally unused memory, otherwise it tends to hamper performance and can reduce the life of your sd card.

Answer 3

Im my case, spdy was the culprit. If I turn it on, the first request works fine, but the next one causes an out of memory though the system has enough memory (less than 50MB are used on a total of 512MB).

Turning it off solved the problem.