Marcelo Tosatti brought this to the attention of vendor-sec
The recent binfmt_aout v2.6 backport changes also fix a DoS:
ChangeSet@1.1527.1.13, 2004-12-16 16:06:31-02:00, firstname.lastname@example.org
[PATCH] a.out: error check on set_brk
It's possible for do_brk() to fail during set_brk() when exec'ing and
a.out. This was noted with Florian's a.out binary and overcommit set to
Capture this error and terminate properly.
ChangeSet@1.1527.1.16, 2004-12-17 21:45:58-02:00, email@example.com
[PATCH] Backport of 2.6 fix to insert_vm_struct to make it return an error
rather than BUG().
Backport of 2.6 fix to insert_vm_struct to make it return an error
rather than BUG(). This eliminates a user triggerable BUG() when user
created a large vma that overlapped with arg pages during exec (could be
triggered with a.out on i386 and x86_64 and elf on ia64).
Signed-off-by: Chris Wright <firstname.lastname@example.org>
Jim, note my comment in bug 144153. I'm not sure if it applies to RHEL2.1.
The second patchset above has already been applied to AS2.1-U7 (see bug 144785).
The first one has not, and is applicable to AS2.1-ia64.
It looks like the first pachset (the do_brk patch) is not applicable to
pensacola nor to derry.
On pensacola (with the binfmt_aout.o kernel module loaded) the test program is
killed with SIGSEGV.
On derry, it appears that we don't ship a binfmt_aout module (correct me if that
The test binary I used is generated as follows:
On code inspection, it looks like the set_brk patch is needed in derry for
binfmt_elf and binfmt_aout, though we don't appear to support aout on derry.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.