Red Hat Bugzilla – Full Text Bug Listing
|Summary:||possible intermittent deadlock with uprobes due to task_finder mm_lock holding|
|Product:||Red Hat Enterprise Linux 5||Reporter:||Frank Ch. Eigler <fche>|
|Component:||systemtap||Assignee:||David Smith <dsmith>|
|Status:||CLOSED ERRATA||QA Contact:||BaseOS QE <qe-baseos-auto>|
|Version:||5.4||CC:||aconway, mjw, pmuller|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2009-09-02 06:01:01 EDT||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
Description Frank Ch. Eigler 2009-06-03 14:44:17 EDT
By code inspection, there is a possibility of deadlock for tasks that undergo a great deal of mm change and are being probed by systemtap user-space probes. We believe this intermittent problem has been seen in the wild both on newer fedora kernels and also on RHEL5 itself. For example, aconway reported this sysrq-t in a state where several user-space processes were hung (unkillable; even ps -awux hung). (aconway, please consider attaching the full log here.) lt-qpidd D ffff810001025a00 0 4798 1 4799 4802 (NOTLB) ffff8104176a5b18 0000000000000086 ffff81043f47c7a0 ffff810435bab0c0 000000c746207980 0000000000000006 ffff81043f47c7a0 ffff81010efa1080 000000c746277f9f 000000000005bea6 ffff81043f47c988 000000040f234380 Call Trace: [<ffffffff8002fd7e>] __up_write+0x27/0xf2 [<ffffffff8006475d>] __down_read+0x7a/0x92 [<ffffffff88536a53>] :uprobes:register_uprobe+0x15d/0x651 [<ffffffff88547c00>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x118/0x20f [<ffffffff88547c25>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:stap_uprobe_change+0x13d/0x20f [<ffffffff88549b6b>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_call_mmap_callbacks+0x72/0x9d [<ffffffff88549d06>] :stap_514b8bf5d340229632a5913932c6e1a7_239797:__stp_utrace_task_finder_target_quiesce+0x170/0x24c [<ffffffff800bcd9c>] report_quiescent+0x39/0x14f [<ffffffff800bceda>] utrace_quiescent+0x28/0x256 [<ffffffff800be188>] utrace_get_signal+0x4e8/0x54b [<ffffffff8002aa6d>] get_signal_to_deliver+0x174/0x45a [<ffffffff8005a794>] do_notify_resume+0x9c/0x7af [<ffffffff800638f0>] schedule_timeout+0x1e/0xad [<ffffffff8002fd7e>] __up_write+0x27/0xf2 [<ffffffff800f2a40>] sys_epoll_wait+0x3b8/0x3f9 [<ffffffff800b4a92>] audit_syscall_exit+0x31b/0x336 [<ffffffff8005d32e>] int_signal+0x12/0x17 There is a fix available that we could apply as a prophylactic even without an easy reproducer: upstream commit 9b59029, 920d63.
Comment 3 David Smith 2009-06-09 15:11:06 EDT
Created attachment 347075 [details] original systemtap script Here's some additional information from aconway. The system was running as part of a cluster. qpidd (a message broker) was running and the perftest program (from qpidc-perftest) was shoving messages through qpidd. I've also attached the systemtap script he was using.
Comment 4 David Smith 2009-06-09 16:15:44 EDT
Created attachment 347084 [details] test source Here's a test program that duplicates the problem from the open posix testsuite (included with ltp). Untar and build using 'make'.
Comment 5 David Smith 2009-06-09 16:22:18 EDT
Created attachment 347085 [details] systemtap script that attaches to the 'stress' program Here's the systemtap script I've used to duplicate the problem. The script is very similar to aconway's original script. The script should be run as: # cd pthread_mutex_lock # stap -v 504007_stress.stp -c ./stress Note that this script was run on an x86_64 system. The path to the 'stress' executable will need to be updated. If the system isn't an x86_64 system, the path to libpthread.so may also need to be adjusted. Note that the deadlock is intermittent. It may help to stress the system by running 'while true; do ps awux > /dev/null; sleep 1; done' in another terminal.
Comment 6 David Smith 2009-06-10 10:10:56 EDT
Created attachment 347225 [details] patch 1 (commit 9b59029)
Comment 7 David Smith 2009-06-10 10:12:01 EDT
Created attachment 347226 [details] patch 2 (commit bec8cf)
Comment 8 David Smith 2009-06-10 10:12:46 EDT
Created attachment 347227 [details] patch 3 (commit 920d63)
Comment 9 David Smith 2009-06-10 10:15:50 EDT
The 3 patches I've attached (with their associated upstream commit ids) seem to fix this problem.
Comment 13 errata-xmlrpc 2009-09-02 06:01:01 EDT
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-1313.html