Bug 239313 - RFE: Document oom_adj & oom_score
Summary: RFE: Document oom_adj & oom_score
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: Deployment_Guide
Version: 5.5
Hardware: All
OS: Linux
Target Milestone: ---
: ---
Assignee: Martin Prpič
QA Contact: ecs-bugs
Depends On:
Blocks: 237606 561593
TreeView+ depends on / blocked
Reported: 2007-05-07 14:59 UTC by Anthony Green
Modified: 2016-06-17 21:05 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2010-08-10 16:35:34 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Anthony Green 2007-05-07 14:59:31 UTC
Description of problem:
The RHEL5 kernel contains a tunable out-of-memory killer algorithm, but the
mechanism used to tune it is undocumented.  See /proc/<PID>/(oom_adj|oom_score).
Please document this so we know how to tune the OOM killer.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Actual results:

Expected results:

Additional info:

Comment 1 Paul Morgan 2007-10-31 14:59:33 UTC
I discovered this tunable a couple months ago and finally got around to adding a
bz request to add tunable to sshd initscript for protection from oom-kill:

What I discovered is that it is only adjustable at /proc/pid/task/pid/oom_adj

Thanks for adding this knob!

Comment 2 Larry Woodman 2008-03-11 15:58:03 UTC
Documantation of oom_adj and oom_score:

The RHEL5 kernel includes 2 files for each process to control when that process
will be considered for termination when the system must start OOM killing.  

/proc/<pid>/oom_adj - Adjust the oom-killer score.
This file can be used to adjust the score used to select which processes shall
be killed in an out-of-memory situation.  Giving a process a high score,
increase the likelihood of this process being killed by the oom-killer. Valid
values are in the range [-16:15], plus the special value '-17', which disables
oom-killing that process altogether.

Example: "echo 15 > proc/<pid>/oom_adj" significantly increase the likelyhood
that process <pid> will be OOM killed.

Example: "echo -16 > proc/<pid>/oom_adj" significantly decrease  the likelyhood
that process <pid> will be OOM killed.

Example: "echo -17 > /proc/<pid>/oom_adj" will disable OOM killing for process
<pid> totally.

Note, the oom score is passed from parent process to child process durring
fork() operations.

/proc/<pid>/oom_score - Display current oom-killer score.
This file can be used to check what the current score used by the oom-killer
for any given <pid>. Use it together with /proc/<pid>/oom_adj to tune which
process will be killed in an out-of-memory situation.

Example: "cat /proc/<pid>/oom_score" will display the current OOM score for
process <pid>.

Comment 4 Don Domingo 2008-06-02 22:36:15 UTC
Hi Linda, Larry,

thanks for the note. as for which book to document this in, i'm thinking
Deployment Guide for now. the Performance Tuning Guide/s are coming along quite
slowly at the moment, seeing as i'm prioritizing release notes over it (we're
currently working on migrating relnotes to publican). 

anyhow, just to verify: are these tunables present as of RHEL5.0 (RHEL5 GA)?
i'll need to know so I can figure out which branches to apply these changes to.

[reassigning bug to me and updating component]


Comment 7 RHEL Program Management 2008-07-10 16:01:43 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update

Comment 8 RHEL Program Management 2008-10-27 18:21:27 UTC
This request was previously evaluated by Red Hat Product Management
for inclusion in the current Red Hat Enterprise Linux release, but
Red Hat was unable to resolve it in time.  This request will be
reviewed for a future Red Hat Enterprise Linux release.

Comment 12 Kazuo Moriwaka 2010-04-08 07:51:18 UTC
Just for your info, kbase has a entry "How do I determine and configure the likelihood that a process will be killed in a out-of-memory situation?"

Comment 13 Martin Prpič 2010-05-27 11:37:30 UTC

here is a proposed solution for this bug, a new section under the "3.3 Directories withing /proc" section. Please let me know if this text will suffice and is correct. Thank you!

3.3.12. /proc/<PID>/

Out of Memory (OOM) refers to a computing state where all available memory, including swap space, has been allocated. Normally this will cause the system to panic and stop functioning as expected. There is a switch that controls OOM behavior in /proc/sys/vm/panic_on_oom. When set to 1 the kernel will panic on OOM. A setting of 0 instructs the kernel to call a function named oom_killer on
an OOM. Usually, oom_killer can kill rogue processes and the system will survive.

The easiest way to change this is to echo the new value to /proc/sys/vm/panic_on_oom.
# cat /proc/sys/vm/panic_on_oom
# echo 0 > /proc/sys/vm/panic_on_oom

# cat /proc/sys/vm/panic_on_oom

It is also possible to prioritize which processes get killed by adjusting the oom_killer score. In /proc/<PID>/ there are two tools labelled oom_adj and oom_score. Valid scores for oom_adj are in the range -16 to +15. To see the current oom_killer score, view the oom_score for the process. oom_killer will kill processes with the highest scores first.

This example adjusts the oom_score of a process with a PID of 12465 to make it less likely that oom_killer will kill it.

# cat /proc/12465/oom_score
# echo -5 > /proc/12465/oom_adj

# cat /proc/12465/oom_score

There is also a special value of -17, which disables oom_killer for that process. In the example below, oom_score returns a value of 0, indicating that this process would not be killed.

# cat /proc/12465/oom_score
# echo -17 > /proc/12465/oom_adj

# cat /proc/12465/oom_score

A function called badness() is used to determine the actual score for each process. This is done by adding up 'points' for each examined process. The process scoring is done in the following way:

1. The basis of each of the process's scores is its memory size.
2. The memory size of any of the process's children (not including a kernel thread) is also added to the score
3. The process's score is increased for 'niced' processes and decreased for long running processes.
4. Processes with the CAP_SYS_ADMIN and CAP_SYS_RAWIO capabilities have their scores reduced.
5. The final score is then bitshifted by the value saved in the oom_adj file.

Thus, a process with the highest oom_score value will most probably be a non-priviliged, recently started process that, along with its children, uses a large amount of memory, has been 'niced', and handles no raw I/O.

Comment 14 Martin Prpič 2010-05-27 12:14:00 UTC
Fixed in RHEL5 trunk, -r31751

Note You need to log in before you can comment on or make changes to this bug.