Bug 504007

Summary: possible intermittent deadlock with uprobes due to task_finder mm_lock holding
Product: Red Hat Enterprise Linux 5 Reporter: Frank Ch. Eigler <fche>
Component: systemtapAssignee: David Smith <dsmith>
Status: CLOSED ERRATA QA Contact: BaseOS QE <qe-baseos-auto>
Severity: high Docs Contact:
Priority: low    
Version: 5.4CC: aconway, mjw, pmuller
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-02 10:01:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
original systemtap script
none
test source
none
systemtap script that attaches to the 'stress' program
none
patch 1 (commit 9b59029)
none
patch 2 (commit bec8cf)
none
patch 3 (commit 920d63) none

Description Frank Ch. Eigler 2009-06-03 18:44:17 UTC 
By code inspection, there is a possibility of deadlock for tasks that
undergo a great deal of mm change and are being probed by systemtap
user-space probes.  We believe this intermittent problem has been
seen in the wild both on newer fedora kernels and also on RHEL5 itself.

For example, aconway reported this sysrq-t in a state where several
user-space processes were hung (unkillable; even ps -awux hung).
(aconway, please consider attaching the full log here.)

lt-qpidd      D ffff810001025a00     0  4798      1          4799  4802 (NOTLB)
ffff8104176a5b18 0000000000000086 ffff81043f47c7a0 ffff810435bab0c0
000000c746207980 0000000000000006 ffff81043f47c7a0 ffff81010efa1080
000000c746277f9f 000000000005bea6 ffff81043f47c988 000000040f234380
Call Trace:
[<ffffffff8002fd7e>] __up_write+0x27/0xf2
[<ffffffff8006475d>] __down_read+0x7a/0x92
[<ffffffff88536a53>] :uprobes:register_uprobe+0x15d/0x651
[<ffffffff88547c00>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x118/0x20f
[<ffffffff88547c25>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x13d/0x20f
[<ffffffff88549b6b>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_call_mmap_callbacks+0x72/0x9d
[<ffffffff88549d06>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_utrace_task_finder_target_quiesce+0x170/0x24c
[<ffffffff800bcd9c>] report_quiescent+0x39/0x14f
[<ffffffff800bceda>] utrace_quiescent+0x28/0x256
[<ffffffff800be188>] utrace_get_signal+0x4e8/0x54b
[<ffffffff8002aa6d>] get_signal_to_deliver+0x174/0x45a
[<ffffffff8005a794>] do_notify_resume+0x9c/0x7af
[<ffffffff800638f0>] schedule_timeout+0x1e/0xad
[<ffffffff8002fd7e>] __up_write+0x27/0xf2
[<ffffffff800f2a40>] sys_epoll_wait+0x3b8/0x3f9
[<ffffffff800b4a92>] audit_syscall_exit+0x31b/0x336
[<ffffffff8005d32e>] int_signal+0x12/0x17


There is a fix available that we could apply as a prophylactic
even without an easy reproducer: upstream commit 9b59029, 920d63.
Comment 3 David Smith 2009-06-09 19:11:06 UTC 
Created attachment 347075 [details]
original systemtap script

Here's some additional information from aconway.  The system was running as part of a cluster.  qpidd (a message broker) was running and the perftest program (from qpidc-perftest) was shoving messages through qpidd.

I've also attached the systemtap script he was using.
Comment 4 David Smith 2009-06-09 20:15:44 UTC 
Created attachment 347084 [details]
test source

Here's a test program that duplicates the problem from the open posix testsuite (included with ltp).  Untar and build using 'make'.
Comment 5 David Smith 2009-06-09 20:22:18 UTC 
Created attachment 347085 [details]
systemtap script that attaches to the 'stress' program

Here's the systemtap script I've used to duplicate the problem.  The script is very similar to aconway's original script.  The script should be run as:

# cd pthread_mutex_lock
# stap -v 504007_stress.stp -c ./stress

Note that this script was run on an x86_64 system.  The path to the 'stress' executable will need to be updated.  If the system isn't an x86_64 system, the path to libpthread.so may also need to be adjusted.

Note that the deadlock is intermittent.  It may help to stress the system by running 'while true; do ps awux > /dev/null; sleep 1; done' in another terminal.
Comment 6 David Smith 2009-06-10 14:10:56 UTC 
Created attachment 347225 [details]
patch 1 (commit 9b59029)
Comment 7 David Smith 2009-06-10 14:12:01 UTC 
Created attachment 347226 [details]
patch 2 (commit bec8cf)
Comment 8 David Smith 2009-06-10 14:12:46 UTC 
Created attachment 347227 [details]
patch 3 (commit 920d63)
Comment 9 David Smith 2009-06-10 14:15:50 UTC 
The 3 patches I've attached (with their associated upstream commit ids) seem to fix this problem.
Comment 13 errata-xmlrpc 2009-09-02 10:01:01 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1313.html