Reported and fixed by Dann Frazier <firstname.lastname@example.org>:
This is an implementation of David Miller's suggested fix in:
Paraphrasing the description from the above report, it makes sendmsg() block while UNIX garbage collection is in progress. This avoids a situation where child processes continue to queue new FDs over a AF_UNIX socket to a parent which is in the exit path and running garbage collection on these FDs. This contention can result in soft lockups and oom-killing of unrelated processes.
This bug was triggerable after fixing CVE-2008-5029. From Dave Miller, "It's a different bug, but triggerable by the same test program."
Patch for this bug: http://marc.info/?l=linux-netdev&m=122771908731133&w=2
Created attachment 324874 [details]
Proposed patch for real-time kernel
This patch is to be applied on top of the CVE-2008-5029 fixes. I have tested it with unix.c in a tight loop, and I did not encounter any soft lock-ups or oom-killer problems. Kindly review/test.
Created attachment 325484 [details]
kernel-220.127.116.11-159.fc10 has been submitted as an update for Fedora 10.
kernel-18.104.22.168-73.fc9 has been submitted as an update for Fedora 9.
kernel-22.214.171.124-159.fc10 has been pushed to the Fedora 10 stable repository. If problems still persist, please make note of it in this bug report.
kernel-126.96.36.199-73.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.
kernel-188.8.131.52-57.fc8 has been pushed to the Fedora 8 stable repository. If problems still persist, please make note of it in this bug report.
Note that for Red Hat Enterprise Linux 5, the patch for CVE-2008-5029 included the fix for this issue. I've updated RHSA-2009:0225 to show that CVE-2008-5300 was addressed.
Created attachment 339407 [details]
Upstream kernel 2.4 patch
This issue has been addressed in following products:
Red Hat Enterprise Linux 3
Via RHSA-2009:1550 https://rhn.redhat.com/errata/RHSA-2009-1550.html