It is possible to render a system with utrace support unusable in ~10 seconds (but not mainline kernel where Ctrl+C will kill process).
Created attachment 149446 [details] fix for utrace/ptrace leak and crash bugs with MT exec This fixes both OOM and BUG_ON failure modes and another crash failure mode from the same problem. I can't reproduce any problems in the test case after this patch, where I got all three failure modes before.
Created attachment 149481 [details] modified test program Here is the version of the test I have been using. It prints out some more info, but not so much that it perturbed the behavior for me.
Fedora bug 227952 is the same problems as this.
Created attachment 149546 [details] fix for utrace/ptrace leak and crash bugs with MT exec + utrace_attach spin failure mode This includes an additional fix that I think resolves a failure mode where utrace_attach spins when you try to kill the test with SIGKILL.
A patch for this issue has been included in zstream build 2.6.18-8.1.2.el5.
fix verified on an x86_64 test system, I had to run 3 concurrent copies of ptrace_thrash to reproduce the problem initially.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2007-0169.html