Red Hat Bugzilla – Bug 504007
possible intermittent deadlock with uprobes due to task_finder mm_lock holding
Last modified: 2009-09-02 06:01:01 EDT
By code inspection, there is a possibility of deadlock for tasks that
undergo a great deal of mm change and are being probed by systemtap
user-space probes. We believe this intermittent problem has been
seen in the wild both on newer fedora kernels and also on RHEL5 itself.
For example, aconway reported this sysrq-t in a state where several
user-space processes were hung (unkillable; even ps -awux hung).
(aconway, please consider attaching the full log here.)
lt-qpidd D ffff810001025a00 0 4798 1 4799 4802 (NOTLB)
ffff8104176a5b18 0000000000000086 ffff81043f47c7a0 ffff810435bab0c0
000000c746207980 0000000000000006 ffff81043f47c7a0 ffff81010efa1080
000000c746277f9f 000000000005bea6 ffff81043f47c988 000000040f234380
There is a fix available that we could apply as a prophylactic
even without an easy reproducer: upstream commit 9b59029, 920d63.
Created attachment 347075 [details]
original systemtap script
Here's some additional information from aconway. The system was running as part of a cluster. qpidd (a message broker) was running and the perftest program (from qpidc-perftest) was shoving messages through qpidd.
I've also attached the systemtap script he was using.
Created attachment 347084 [details]
Here's a test program that duplicates the problem from the open posix testsuite (included with ltp). Untar and build using 'make'.
Created attachment 347085 [details]
systemtap script that attaches to the 'stress' program
Here's the systemtap script I've used to duplicate the problem. The script is very similar to aconway's original script. The script should be run as:
# cd pthread_mutex_lock
# stap -v 504007_stress.stp -c ./stress
Note that this script was run on an x86_64 system. The path to the 'stress' executable will need to be updated. If the system isn't an x86_64 system, the path to libpthread.so may also need to be adjusted.
Note that the deadlock is intermittent. It may help to stress the system by running 'while true; do ps awux > /dev/null; sleep 1; done' in another terminal.
Created attachment 347225 [details]
patch 1 (commit 9b59029)
Created attachment 347226 [details]
patch 2 (commit bec8cf)
Created attachment 347227 [details]
patch 3 (commit 920d63)
The 3 patches I've attached (with their associated upstream commit ids) seem to fix this problem.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.