fs/exec.c in Linux 2.6, when one thread is tracing another thread that shares the same memory map, might allow local users to cause a denial of service (deadlock) by forcing a core dump when the traced thread is in the TASK_TRACED state. http://linux.bkbits.net:8080/linux-2.6/cset%4041e9a97cuQ7FWekabtf12Orvpfbp1w This issue does not affect linux-2.4 but may affect RHEL3 due to the backported nptl patch in linux-2.4.20-o1-nptl.patch. (Note that we fixed this in RHEL4U1 in *6.9-ptrace-fixes.patch)
Reassigning to PeterS at Linda's request.
RHEL3 does not have the TASK_TRACED state (only TASK_STOPPED), so the failure mode is not exactly the same in that regard. I will dig up the test program that reproduces the problem on affected kernels, and then we can see what it does on RHEL3.
Created attachment 120824 [details] Testcase to reproduce the problem
The patch refered to above in the bitkeeper bits is not sufficient to address this situation in RHEL-3. A hang still occurs when the test program is run. Some more diagnosis needs to be done in order to discover what the situation is and what needs to be done to address it.
Peter wrote "Mark, the impact to the system for this issue seems to be small. A user can hang his own process, but will not be able to create more processes than could otherwise be created. The system remains functional while the the process is hanging, so the possibility of an DoS attack, using this situation, seems minimal." Reducing to low severity
Created attachment 124114 [details] Proposed patch
There were two sets of changes required in order to address the issue here. One was to keep the threads in the thread group, which are being destroyed, from issuing SIGCHLD to their parent and waiting for it to reap them. The second was to correct the parent handling in the task struct in order to prevent a child from attempting to become its own parent.
Created attachment 124115 [details] Test program to reproduce the situation.
This issue is on Red Hat Engineering's list of planned work items for the upcoming Red Hat Enterprise Linux 3.8 release. Engineering resources have been assigned and barring unforeseen circumstances, Red Hat intends to include this item in the 3.8 release.
A fix for this problem has just been committed to the RHEL3 U8 patch pool this evening (in kernel version 2.4.21-40.10.EL).
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2006-0437.html