Marcelo Tosatti brought this to the attention of vendor-sec The recent binfmt_aout v2.6 backport changes also fix a DoS: ChangeSet.1.13, 2004-12-16 16:06:31-02:00, chrisw [PATCH] a.out: error check on set_brk It's possible for do_brk() to fail during set_brk() when exec'ing and a.out. This was noted with Florian's a.out binary and overcommit set to 0. Capture this error and terminate properly. ChangeSet.1.16, 2004-12-17 21:45:58-02:00, chrisw [PATCH] Backport of 2.6 fix to insert_vm_struct to make it return an error rather than BUG(). Backport of 2.6 fix to insert_vm_struct to make it return an error rather than BUG(). This eliminates a user triggerable BUG() when user created a large vma that overlapped with arg pages during exec (could be triggered with a.out on i386 and x86_64 and elf on ia64). Signed-off-by: Chris Wright <chrisw> This issue should also affect FC2
A second look tells me we're OK.