When does the OS kill an application?

Question

I have an application which polls a bunch of servers every few minutes. To do this, it spawns one thread per server to poll (15 servers) and writes back the data to an object, however sometimes the threads do not exit and stay 'stuck', consuming memory. After about 150-200 threads are stuck, the process is killed by the OS. This is what I get from running the application with time from the CLI:

166 active threads.
167 active threads.
Killed

real    247m45.277s
user    226m22.750s
sys     13m3.150s
$

Here are some typical ps aux outputs for the application:

USER       PID %CPU %MEM    VSZ   RSS   TTY    STAT START   TIME COMMAND
pi       21998 69.0  3.1  86964 15592   pts/3  Sl+  19:46   0:04 python app.py

USER       PID %CPU %MEM    VSZ   RSS   TTY    STAT START   TIME COMMAND
pi       21998 95.8  3.5  72488 17596   pts/3  Sl+  19:46   0:53 python app.py

USER       PID %CPU %MEM    VSZ   RSS   TTY    STAT START   TIME COMMAND
pi       21998 97.6  4.3 101280 21800   pts/3  Sl+  19:46   2:31 python app.py

USER       PID %CPU %MEM    VSZ    RSS  TTY    STAT START   TIME COMMAND
pi       26077 97.7 22.7 429224 113148  pts/2  Sl+  13:54  53:34 python app.py

USER       PID %CPU %MEM    VSZ    RSS  TTY    STAT START   TIME COMMAND
pi        2543 98.4 61.9 1098460 308272 pts/2  Sl+  08:12 120:17 python bunny.py

I have seen the RSS value go up past 200000 KiB after the app has been running for a few hours, but I don't have the ps output for any of those runs (I'm running one now).

At what point does the OS kill the application? Is it a memory limit? Is it a CPU limit? Where is this set?

Answer 1

You can check the kernel log:

/var/log/kern.log

If the kernel killed a process due to a lack of system resources it should be logged in the above file.

Answer 2

The Linux kernel "overcommits" memory, that is when your process allocates virtual memory it does not immediately allocate physical memory or swap space. The physical memory is only allocated when the application actually uses the memory. Most of the time this is a good thing, appliations frequently allocate far more memory than they actually use.

Initially as applications need memory stuff will be pushed out of caches or pushed to swap (if you have it). But there comes a point at which there is simply no memory left to free up by those methods.

When that happens the OOM killer comes into play. It goes through the list of processes looking for a victim. It has heuristics to determine which process is the "best" target for killing. https://linux-mm.org/OOM_Killer

Answer 3

when os kills your application, that means your application misbehaves and have to be fixed before trying to adjust some obscure system settings that will only help to mask the problem and make it even more difficult to diagnose.

it's very likely that your threads get stuck on network timeouts, and you'd better add to your python script in the very beginning:

import socket
socket.setdefaulttimeout( 10 )  # timeout in seconds