Bug 239313

Summary: RFE: Document oom_adj & oom_score
Product: Red Hat Enterprise Linux 5 Reporter: Anthony Green <green>
Component: doc-Deployment_GuideAssignee: Martin Prpič <mprpic>
Status: CLOSED CURRENTRELEASE QA Contact: ecs-bugs
Severity: medium Docs Contact:
Priority: medium    
Version: 5.5CC: cwyse, ddomingo, jskeoch, kmoriwak, lwoodman, pmorgan, riek
Target Milestone: ---Keywords: Documentation
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-08-10 16:35:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 237606, 561593    

Description Anthony Green 2007-05-07 14:59:31 UTC
Description of problem:
The RHEL5 kernel contains a tunable out-of-memory killer algorithm, but the
mechanism used to tune it is undocumented.  See /proc/<PID>/(oom_adj|oom_score).
Please document this so we know how to tune the OOM killer.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Paul Morgan 2007-10-31 14:59:33 UTC
I discovered this tunable a couple months ago and finally got around to adding a
bz request to add tunable to sshd initscript for protection from oom-kill:
https://bugzilla.redhat.com/show_bug.cgi?id=360341

What I discovered is that it is only adjustable at /proc/pid/task/pid/oom_adj

Thanks for adding this knob!

Comment 2 Larry Woodman 2008-03-11 15:58:03 UTC
Documantation of oom_adj and oom_score:

The RHEL5 kernel includes 2 files for each process to control when that process
will be considered for termination when the system must start OOM killing.  

/proc/<pid>/oom_adj - Adjust the oom-killer score.
This file can be used to adjust the score used to select which processes shall
be killed in an out-of-memory situation.  Giving a process a high score,
increase the likelihood of this process being killed by the oom-killer. Valid
values are in the range [-16:15], plus the special value '-17', which disables
oom-killing that process altogether.

Example: "echo 15 > proc/<pid>/oom_adj" significantly increase the likelyhood
that process <pid> will be OOM killed.

Example: "echo -16 > proc/<pid>/oom_adj" significantly decrease  the likelyhood
that process <pid> will be OOM killed.

Example: "echo -17 > /proc/<pid>/oom_adj" will disable OOM killing for process
<pid> totally.

Note, the oom score is passed from parent process to child process durring
fork() operations.


/proc/<pid>/oom_score - Display current oom-killer score.
This file can be used to check what the current score used by the oom-killer
for any given <pid>. Use it together with /proc/<pid>/oom_adj to tune which
process will be killed in an out-of-memory situation.

Example: "cat /proc/<pid>/oom_score" will display the current OOM score for
process <pid>.

Comment 4 Don Domingo 2008-06-02 22:36:15 UTC
Hi Linda, Larry,

thanks for the note. as for which book to document this in, i'm thinking
Deployment Guide for now. the Performance Tuning Guide/s are coming along quite
slowly at the moment, seeing as i'm prioritizing release notes over it (we're
currently working on migrating relnotes to publican). 

anyhow, just to verify: are these tunables present as of RHEL5.0 (RHEL5 GA)?
i'll need to know so I can figure out which branches to apply these changes to.

[reassigning bug to me and updating component]

thanks!

Comment 7 RHEL Program Management 2008-07-10 16:01:43 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 8 RHEL Program Management 2008-10-27 18:21:27 UTC
This request was previously evaluated by Red Hat Product Management
for inclusion in the current Red Hat Enterprise Linux release, but
Red Hat was unable to resolve it in time.  This request will be
reviewed for a future Red Hat Enterprise Linux release.

Comment 12 Kazuo Moriwaka 2010-04-08 07:51:18 UTC
Just for your info, kbase has a entry "How do I determine and configure the likelihood that a process will be killed in a out-of-memory situation?"
http://kbase.redhat.com/faq/docs/DOC-5359

Comment 13 Martin Prpič 2010-05-27 11:37:30 UTC
Larry, 

here is a proposed solution for this bug, a new section under the "3.3 Directories withing /proc" section. Please let me know if this text will suffice and is correct. Thank you!

<SNIP>
3.3.12. /proc/<PID>/

Out of Memory (OOM) refers to a computing state where all available memory, including swap space, has been allocated. Normally this will cause the system to panic and stop functioning as expected. There is a switch that controls OOM behavior in /proc/sys/vm/panic_on_oom. When set to 1 the kernel will panic on OOM. A setting of 0 instructs the kernel to call a function named oom_killer on
an OOM. Usually, oom_killer can kill rogue processes and the system will survive.

The easiest way to change this is to echo the new value to /proc/sys/vm/panic_on_oom.
                                                                                
# cat /proc/sys/vm/panic_on_oom
1
# echo 0 > /proc/sys/vm/panic_on_oom

# cat /proc/sys/vm/panic_on_oom
0

It is also possible to prioritize which processes get killed by adjusting the oom_killer score. In /proc/<PID>/ there are two tools labelled oom_adj and oom_score. Valid scores for oom_adj are in the range -16 to +15. To see the current oom_killer score, view the oom_score for the process. oom_killer will kill processes with the highest scores first.

This example adjusts the oom_score of a process with a PID of 12465 to make it less likely that oom_killer will kill it.

# cat /proc/12465/oom_score
79872
# echo -5 > /proc/12465/oom_adj

# cat /proc/12465/oom_score
78

There is also a special value of -17, which disables oom_killer for that process. In the example below, oom_score returns a value of 0, indicating that this process would not be killed.

# cat /proc/12465/oom_score
78
# echo -17 > /proc/12465/oom_adj

# cat /proc/12465/oom_score
0

A function called badness() is used to determine the actual score for each process. This is done by adding up 'points' for each examined process. The process scoring is done in the following way:

1. The basis of each of the process's scores is its memory size.
2. The memory size of any of the process's children (not including a kernel thread) is also added to the score
3. The process's score is increased for 'niced' processes and decreased for long running processes.
4. Processes with the CAP_SYS_ADMIN and CAP_SYS_RAWIO capabilities have their scores reduced.
5. The final score is then bitshifted by the value saved in the oom_adj file.

Thus, a process with the highest oom_score value will most probably be a non-priviliged, recently started process that, along with its children, uses a large amount of memory, has been 'niced', and handles no raw I/O.
</SNIP>

Comment 14 Martin Prpič 2010-05-27 12:14:00 UTC
Fixed in RHEL5 trunk, -r31751