Red Hat Bugzilla – Bug 144362
CAN-2004-1074 a.out binfmt DoS
Last modified: 2007-11-30 17:06:54 EST
*** This bug has been split off bug 144361 *** ------- Original comment by Josh Bressers (Security Response Team) on 2005.01.06 10:21 ------- From the lkml mailing list. http://www.ussg.iu.edu/hypermail/linux/kernel/0411.1/1222.html It is possible that an improperly formed a.out binary can cause a kernel-oops, which if executed in a loop will ead fd's and memory. It seems that you have to turn on memory overcommit on for this to work. davej says we didn't enable a.out binaries on FC3 and RHEL4. This will be undone in the future though. We do not turn on memory overcommit by default in RHEL2.1 or 3. This should make the impact of this issue significantly mitigated since it seems the issue is only exploitable when overcommit is on (sysctl -w vm.overcommit_memory=1). However note that we do tell customers how to turn this on. fixed=2.6.10 (20041116 cset@419aaba8xdR0decwoMnVpt3G8_f8kQ) not fixed for 2.4 (as at Nov24) Patch: http://www.ussg.iu.edu/hypermail/linux/kernel/0411.1/1290.html
This is NOTABUG for AS2.1-i386 since it is based on 2.4.9 which handles vma's differently. The poisoned binary exits with an error rather than crashing, so there's no DoS.