Bug 504007 - possible intermittent deadlock with uprobes due to task_finder mm_lock holding
possible intermittent deadlock with uprobes due to task_finder mm_lock holding
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: systemtap (Show other bugs)
All Linux
low Severity high
: rc
: ---
Assigned To: David Smith
Depends On:
  Show dependency treegraph
Reported: 2009-06-03 14:44 EDT by Frank Ch. Eigler
Modified: 2009-09-02 06:01 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2009-09-02 06:01:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
original systemtap script (830 bytes, text/plain)
2009-06-09 15:11 EDT, David Smith
no flags Details
test source (16.06 KB, application/x-bzip2)
2009-06-09 16:15 EDT, David Smith
no flags Details
systemtap script that attaches to the 'stress' program (539 bytes, text/plain)
2009-06-09 16:22 EDT, David Smith
no flags Details
patch 1 (commit 9b59029) (5.32 KB, patch)
2009-06-10 10:10 EDT, David Smith
no flags Details | Diff
patch 2 (commit bec8cf) (6.06 KB, patch)
2009-06-10 10:12 EDT, David Smith
no flags Details | Diff
patch 3 (commit 920d63) (928 bytes, patch)
2009-06-10 10:12 EDT, David Smith
no flags Details | Diff

  None (edit)
Description Frank Ch. Eigler 2009-06-03 14:44:17 EDT
By code inspection, there is a possibility of deadlock for tasks that
undergo a great deal of mm change and are being probed by systemtap
user-space probes.  We believe this intermittent problem has been
seen in the wild both on newer fedora kernels and also on RHEL5 itself.

For example, aconway reported this sysrq-t in a state where several
user-space processes were hung (unkillable; even ps -awux hung).
(aconway, please consider attaching the full log here.)

lt-qpidd      D ffff810001025a00     0  4798      1          4799  4802 (NOTLB)
ffff8104176a5b18 0000000000000086 ffff81043f47c7a0 ffff810435bab0c0
000000c746207980 0000000000000006 ffff81043f47c7a0 ffff81010efa1080
000000c746277f9f 000000000005bea6 ffff81043f47c988 000000040f234380
Call Trace:
[<ffffffff8002fd7e>] __up_write+0x27/0xf2
[<ffffffff8006475d>] __down_read+0x7a/0x92
[<ffffffff88536a53>] :uprobes:register_uprobe+0x15d/0x651
[<ffffffff88547c00>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x118/0x20f
[<ffffffff88547c25>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x13d/0x20f
[<ffffffff88549b6b>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_call_mmap_callbacks+0x72/0x9d
[<ffffffff88549d06>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_utrace_task_finder_target_quiesce+0x170/0x24c
[<ffffffff800bcd9c>] report_quiescent+0x39/0x14f
[<ffffffff800bceda>] utrace_quiescent+0x28/0x256
[<ffffffff800be188>] utrace_get_signal+0x4e8/0x54b
[<ffffffff8002aa6d>] get_signal_to_deliver+0x174/0x45a
[<ffffffff8005a794>] do_notify_resume+0x9c/0x7af
[<ffffffff800638f0>] schedule_timeout+0x1e/0xad
[<ffffffff8002fd7e>] __up_write+0x27/0xf2
[<ffffffff800f2a40>] sys_epoll_wait+0x3b8/0x3f9
[<ffffffff800b4a92>] audit_syscall_exit+0x31b/0x336
[<ffffffff8005d32e>] int_signal+0x12/0x17

There is a fix available that we could apply as a prophylactic
even without an easy reproducer: upstream commit 9b59029, 920d63.
Comment 3 David Smith 2009-06-09 15:11:06 EDT
Created attachment 347075 [details]
original systemtap script

Here's some additional information from aconway.  The system was running as part of a cluster.  qpidd (a message broker) was running and the perftest program (from qpidc-perftest) was shoving messages through qpidd.

I've also attached the systemtap script he was using.
Comment 4 David Smith 2009-06-09 16:15:44 EDT
Created attachment 347084 [details]
test source

Here's a test program that duplicates the problem from the open posix testsuite (included with ltp).  Untar and build using 'make'.
Comment 5 David Smith 2009-06-09 16:22:18 EDT
Created attachment 347085 [details]
systemtap script that attaches to the 'stress' program

Here's the systemtap script I've used to duplicate the problem.  The script is very similar to aconway's original script.  The script should be run as:

# cd pthread_mutex_lock
# stap -v 504007_stress.stp -c ./stress

Note that this script was run on an x86_64 system.  The path to the 'stress' executable will need to be updated.  If the system isn't an x86_64 system, the path to libpthread.so may also need to be adjusted.

Note that the deadlock is intermittent.  It may help to stress the system by running 'while true; do ps awux > /dev/null; sleep 1; done' in another terminal.
Comment 6 David Smith 2009-06-10 10:10:56 EDT
Created attachment 347225 [details]
patch 1 (commit 9b59029)
Comment 7 David Smith 2009-06-10 10:12:01 EDT
Created attachment 347226 [details]
patch 2 (commit bec8cf)
Comment 8 David Smith 2009-06-10 10:12:46 EDT
Created attachment 347227 [details]
patch 3 (commit 920d63)
Comment 9 David Smith 2009-06-10 10:15:50 EDT
The 3 patches I've attached (with their associated upstream commit ids) seem to fix this problem.
Comment 13 errata-xmlrpc 2009-09-02 06:01:01 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.