Marcelo Tosatti brought this to the attention of vendor-sec The recent binfmt_aout v2.6 backport changes also fix a DoS: ChangeSet.1.13, 2004-12-16 16:06:31-02:00, chrisw [PATCH] a.out: error check on set_brk It's possible for do_brk() to fail during set_brk() when exec'ing and a.out. This was noted with Florian's a.out binary and overcommit set to 0. Capture this error and terminate properly. ChangeSet.1.16, 2004-12-17 21:45:58-02:00, chrisw [PATCH] Backport of 2.6 fix to insert_vm_struct to make it return an error rather than BUG(). Backport of 2.6 fix to insert_vm_struct to make it return an error rather than BUG(). This eliminates a user triggerable BUG() when user created a large vma that overlapped with arg pages during exec (could be triggered with a.out on i386 and x86_64 and elf on ia64). Signed-off-by: Chris Wright <chrisw>
While I agree that we should fix such a problem in RHEL3, support for a.out format binaries is disabled by default (in RHEL3), and thus users cannot crash RHEL3 kernels over a.out handling bugs under normal circumstances. Lowering priority to NORMAL.
I've determined that this was fixed in U5 (in kernel version 2.4.21-27.8.EL) as well as in the 2nd E5 build (in kernel version 2.4.21-27.0.2.EL). Here is the comment associated with the release of that E5 advisory: "An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2005-043.html" Obviously, we'd rather have people upgrade to U5 at this point, which was released with advisory RHSA-2005:294. *** This bug has been marked as a duplicate of 144361 ***