Bug 750399 (CVE-2011-4097) - CVE-2011-4097 kernel: oom_badness() integer overflow
Summary: CVE-2011-4097 kernel: oom_badness() integer overflow
Alias: CVE-2011-4097
Product: Security Response
Classification: Other
Component: vulnerability
Version: unspecified
Hardware: All
OS: Linux
Target Milestone: ---
Assignee: Red Hat Product Security
QA Contact:
Whiteboard: impact=moderate,public=20111031,repor...
Keywords: Security
Depends On: 741207 750401 750402 761376
Blocks: 747636
TreeView+ depends on / blocked
Reported: 2011-11-01 00:36 UTC by Eugene Teo (Security Response)
Modified: 2019-06-08 18:57 UTC (History)
22 users (show)

Clone Of:
Last Closed: 2012-05-04 08:05:08 UTC

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:0333 normal SHIPPED_LIVE Important: kernel-rt security and bug fix update 2012-02-24 01:21:35 UTC

Description Eugene Teo (Security Response) 2011-11-01 00:36:33 UTC
An integer overflow will happen on 64bit archs if task's sum of rss, swapents and nr_ptes exceeds (2^31)/1000 value. This was introduced by commit f755a04 oom: use pte pages in OOM score

where the oom score computation was divided into several steps and it's no longer computed as one expression in unsigned long(rss, swapents, nr_pte are unsigned long), where the result value assigned to points(int) is in range(1..1000). So there could be an int overflow while computing

176          points *= 1000;

and points may have negative value. Meaning the oom score for a mem hog task will be one.

196          if (points <= 0)
197                  return 1;
For example:
[ 3366]     0  3366 35390480 24303939   5       0             0 oom01
Out of memory: Kill process 3366 (oom01) score 1 or sacrifice child

Here the oom1 process consumes more than 24303939(rss)*4096~=92GB physical memory, but it's oom score is one.

In this situation the mem hog task is skipped and oom killer kills another and most probably innocent task with oom score greater than one.

The points variable should be of type long instead of int to prevent the int overflow.

Signed-off-by: Frantisek Hrbata <fhrbata@redhat.com>
 mm/oom_kill.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 626303b..e9a1785 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -162,7 +162,7 @@ static bool oom_unkillable_task(struct task_struct *p,
 unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *mem,
 		      const nodemask_t *nodemask, unsigned long totalpages)
-	int points;
+	long points;
 	if (oom_unkillable_task(p, mem, nodemask))
 		return 0;

Introduced by:

Upstream commit:


Red Hat would like to thank Shubham Goyal for reporting this issue.

Comment 2 Eugene Teo (Security Response) 2011-11-01 00:41:49 UTC
Created kernel tracking bugs for this issue

Affects: fedora-all [bug 750402]

Comment 3 Eugene Teo (Security Response) 2011-11-01 00:45:24 UTC

This issue did not affect the Linux kernel as shipped with Red Hat Enterprise Linux 4, 5, and 6 as they did not backport the upstream commit f755a04 that introduced this. This has been addressed in Red Hat Enterprise MRG via https://rhn.redhat.com/errata/RHSA-2012-0333.html.

Comment 6 errata-xmlrpc 2012-02-23 20:23:12 UTC
This issue has been addressed in following products:

  MRG for RHEL-6 v.2

Via RHSA-2012:0333 https://rhn.redhat.com/errata/RHSA-2012-0333.html

Note You need to log in before you can comment on or make changes to this bug.