By code inspection, there is a possibility of deadlock for tasks that undergo a great deal of mm change and are being probed by systemtap user-space probes. We believe this intermittent problem has been seen in the wild both on newer fedora kernels and also on RHEL5 itself. For example, aconway reported this sysrq-t in a state where several user-space processes were hung (unkillable; even ps -awux hung). (aconway, please consider attaching the full log here.) lt-qpidd D ffff810001025a00 0 4798 1 4799 4802 (NOTLB) ffff8104176a5b18 0000000000000086 ffff81043f47c7a0 ffff810435bab0c0 000000c746207980 0000000000000006 ffff81043f47c7a0 ffff81010efa1080 000000c746277f9f 000000000005bea6 ffff81043f47c988 000000040f234380 Call Trace: [<ffffffff8002fd7e>] __up_write+0x27/0xf2 [<ffffffff8006475d>] __down_read+0x7a/0x92 [<ffffffff88536a53>] :uprobes:register_uprobe+0x15d/0x651 [<ffffffff88547c00>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x118/0x20f [<ffffffff88547c25>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x13d/0x20f [<ffffffff88549b6b>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_call_mmap_callbacks+0x72/0x9d [<ffffffff88549d06>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_utrace_task_finder_target_quiesce+0x170/0x24c [<ffffffff800bcd9c>] report_quiescent+0x39/0x14f [<ffffffff800bceda>] utrace_quiescent+0x28/0x256 [<ffffffff800be188>] utrace_get_signal+0x4e8/0x54b [<ffffffff8002aa6d>] get_signal_to_deliver+0x174/0x45a [<ffffffff8005a794>] do_notify_resume+0x9c/0x7af [<ffffffff800638f0>] schedule_timeout+0x1e/0xad [<ffffffff8002fd7e>] __up_write+0x27/0xf2 [<ffffffff800f2a40>] sys_epoll_wait+0x3b8/0x3f9 [<ffffffff800b4a92>] audit_syscall_exit+0x31b/0x336 [<ffffffff8005d32e>] int_signal+0x12/0x17 There is a fix available that we could apply as a prophylactic even without an easy reproducer: upstream commit 9b59029, 920d63.
Created attachment 347075 [details] original systemtap script Here's some additional information from aconway. The system was running as part of a cluster. qpidd (a message broker) was running and the perftest program (from qpidc-perftest) was shoving messages through qpidd. I've also attached the systemtap script he was using.
Created attachment 347084 [details] test source Here's a test program that duplicates the problem from the open posix testsuite (included with ltp). Untar and build using 'make'.
Created attachment 347085 [details] systemtap script that attaches to the 'stress' program Here's the systemtap script I've used to duplicate the problem. The script is very similar to aconway's original script. The script should be run as: # cd pthread_mutex_lock # stap -v 504007_stress.stp -c ./stress Note that this script was run on an x86_64 system. The path to the 'stress' executable will need to be updated. If the system isn't an x86_64 system, the path to libpthread.so may also need to be adjusted. Note that the deadlock is intermittent. It may help to stress the system by running 'while true; do ps awux > /dev/null; sleep 1; done' in another terminal.
Created attachment 347225 [details] patch 1 (commit 9b59029)
Created attachment 347226 [details] patch 2 (commit bec8cf)
Created attachment 347227 [details] patch 3 (commit 920d63)
The 3 patches I've attached (with their associated upstream commit ids) seem to fix this problem.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-1313.html