2

I have an application which polls a bunch of servers every few minutes. To do this, it spawns one thread per server to poll (15 servers) and writes back the data to an object, however sometimes the threads do not exit and stay 'stuck', consuming memory. After about 150-200 threads are stuck, the process is killed by the OS. This is what I get from running the application with time from the CLI:

166 active threads.
167 active threads.
Killed

real    247m45.277s
user    226m22.750s
sys     13m3.150s
$

Here are some typical ps aux outputs for the application:

USER       PID %CPU %MEM    VSZ   RSS   TTY    STAT START   TIME COMMAND
pi       21998 69.0  3.1  86964 15592   pts/3  Sl+  19:46   0:04 python app.py

USER       PID %CPU %MEM    VSZ   RSS   TTY    STAT START   TIME COMMAND
pi       21998 95.8  3.5  72488 17596   pts/3  Sl+  19:46   0:53 python app.py

USER       PID %CPU %MEM    VSZ   RSS   TTY    STAT START   TIME COMMAND
pi       21998 97.6  4.3 101280 21800   pts/3  Sl+  19:46   2:31 python app.py

USER       PID %CPU %MEM    VSZ    RSS  TTY    STAT START   TIME COMMAND
pi       26077 97.7 22.7 429224 113148  pts/2  Sl+  13:54  53:34 python app.py

USER       PID %CPU %MEM    VSZ    RSS  TTY    STAT START   TIME COMMAND
pi        2543 98.4 61.9 1098460 308272 pts/2  Sl+  08:12 120:17 python bunny.py

I have seen the RSS value go up past 200000 KiB after the app has been running for a few hours, but I don't have the ps output for any of those runs (I'm running one now).

At what point does the OS kill the application? Is it a memory limit? Is it a CPU limit? Where is this set?

3

You can check the kernel log:

/var/log/kern.log

If the kernel killed a process due to a lack of system resources it should be logged in the above file.

  • Thank you. Sure enough, the application is being killed due to memory usage. This, despite the >95% CPU usage! – dotancohen Aug 2 '13 at 9:07
  • Just so that the answer could be complete and that I would be able to accept it: Where is the config setting for how much memory an application can hog before it is killed? Or, how could I check what that value is without hitting it empirically? – dotancohen Aug 2 '13 at 9:09
  • 1
    @dotancohen When some process tries to request memory and there is no memory available, the Linux kernel starts killing (other) processes. The priority can be controlled with the /proc/self/oom_score_adj (see also the manual page of proc). You can also consider adding swap to extend the memory, but note that this is likely going to hurt performance due to slow I/O. I'd suggest to solve the problem at the root, why does the memory usage peek? – Lekensteyn Aug 2 '13 at 9:45
  • Thank you. I'm right now using the opportunity to figure out how the OS works. now that I know, I'll get to the stage of fixing the application! I don't want to actually change the OS settings, just to understand them. – dotancohen Aug 2 '13 at 9:53
2

The Linux kernel "overcommits" memory, that is when your process allocates virtual memory it does not immediately allocate physical memory or swap space. The physical memory is only allocated when the application actually uses the memory. Most of the time this is a good thing, appliations frequently allocate far more memory than they actually use.

Initially as applications need memory stuff will be pushed out of caches or pushed to swap (if you have it). But there comes a point at which there is simply no memory left to free up by those methods.

When that happens the OOM killer comes into play. It goes through the list of processes looking for a victim. It has heuristics to determine which process is the "best" target for killing. https://linux-mm.org/OOM_Killer

1

when os kills your application, that means your application misbehaves and have to be fixed before trying to adjust some obscure system settings that will only help to mask the problem and make it even more difficult to diagnose.

it's very likely that your threads get stuck on network timeouts, and you'd better add to your python script in the very beginning:

import socket
socket.setdefaulttimeout( 10 )  # timeout in seconds
  • Thank you. Though I also thought that the threads are getting stuck on network timeouts, added the default timeout as you suggest does not help. – dotancohen Aug 2 '13 at 7:57
  • Note that at this stage I'm using the opportunity to learn about Linux. now that I know how it works and when it kills applications, I'll get to fixing the app. Thank you! – dotancohen Aug 2 '13 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.