Description of problem: This issue was mentioned in http://grsecurity.net/~spender/64bit_dos.c. Written in the comments: "The second bug here is that the memory usage explodes within the kernel from a single 128k allocation in userland The explosion of memory isn't accounted for by any task so it won't be terminated by the OOM killer." Acknowledgements: Red Hat would like to thank Brad Spengler for reporting this issue.
Two issues here, the BUG_ON condition and the OOM dodging issue. Roland proposed the solution to the BUG_ON issue with http://lkml.org/lkml/2010/8/30/463 as opposed to Kee's http://www.openwall.com/lists/oss-security/2010/08/27/1. And Motohiro-san proposed http://lkml.org/lkml/2010/8/29/206 for the OOM dodging issue, but no feedback yet. re: reproducer, Alexander noted http://lkml.org/lkml/2010/8/30/378. So I see two possible two CVE assignments.
Introduced by upstream commit b6a2fea39318e43fee84fa7b0b90d68bed92d2ba. For my reference, bug 443659 (rhel-5).
The top-level bug for the BUG_ON issue is bug 645222. This bug will be used to address the OOM dodging issue.
Update: http://lkml.org/lkml/2010/10/24/207
(In reply to comment #15) > Update: > http://lkml.org/lkml/2010/10/24/207 http://linux.derkeiler.com/Mailing-Lists/Kernel/2010-11/msg13278.html
(In reply to comment #16) > (In reply to comment #15) > > Update: > > http://lkml.org/lkml/2010/10/24/207 > > http://linux.derkeiler.com/Mailing-Lists/Kernel/2010-11/msg13278.html upstream commit 3c77f845722158206a7209c45ccddc264d19319c
This issue has been addressed in following products: Red Hat Enterprise Linux 5 Via RHSA-2011:0017 https://rhn.redhat.com/errata/RHSA-2011-0017.html
This issue has been addressed in following products: Red Hat Enterprise Linux 6 Via RHSA-2011:0283 https://rhn.redhat.com/errata/RHSA-2011-0283.html
This issue has been addressed in following products: MRG for RHEL-6 v.2 Via RHSA-2011:1253 https://rhn.redhat.com/errata/RHSA-2011-1253.html