Bug 859009 - libvirtd deadlocks itself from time to time via virDomainDestroy
libvirtd deadlocks itself from time to time via virDomainDestroy
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
18
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Libvirt Maintainers
Fedora Extras Quality Assurance
:
: 856639 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-20 07:00 EDT by Richard W.M. Jones
Modified: 2012-11-28 09:56 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-28 09:56:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Richard W.M. Jones 2012-09-20 07:00:41 EDT
Description of problem:

I don't have a way to reproduce this, but every so often I
come to run some libguestfs command and find that it blocks
trying to connect to libvirtd.  libvirtd (non-root, qemu:///session)
deadlocks itself.

Version-Release number of selected component (if applicable):

libvirt from git (5090c576e3a76727c39a9b3221b2e82b545bc983)
Fedora 18

How reproducible:

It's happened a handful of times now, but I don't have a way to
reproduce it.

Steps to Reproduce:
<unknown>
  
Actual results:
0x00000030d0e0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install cryptopp-5.6.1-7.fc17.x86_64 glibc-2.16-3.fc18.x86_64 gnutls-2.12.20-1.fc18.x86_64 libcom_err-1.42.4-1.fc18.x86_64 libcurl-7.26.0-4.fc18.x86_64 libdb-5.3.15-3.fc18.x86_64 libgcrypt-1.5.0-5.fc18.x86_64 libidn-1.25-2.fc18.x86_64 libpcap-1.3.0-1.fc18.x86_64 libpciaccess-0.13.1-1.fc18.x86_64 libselinux-2.1.11-1.fc18.x86_64 libsepol-2.1.7-1.fc18.x86_64 libssh2-1.4.2-1.fc18.x86_64 libtasn1-2.13-1.fc18.x86_64 libwsman1-2.3.0-3.fc18.x86_64 netcf-libs-0.2.2-1.fc19.x86_64 nspr-4.9.1-4.fc18.x86_64 nss-3.13.5-5.fc18.x86_64 nss-softokn-freebl-3.13.5-2.fc18.x86_64 nss-util-3.13.5-3.fc18.x86_64 numactl-libs-2.0.7-6.fc18.x86_64 xen-libs-4.1.2-20.fc18.x86_64 xz-libs-5.1.2-1alpha.fc18.x86_64 zlib-1.2.7-3.fc18.x86_64
(gdb) t a a bt

Thread 18 (Thread 0x7fcd15e28700 (LWP 26745)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe0d090)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fcd15627700 (LWP 26746)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe0d390)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fcd14e26700 (LWP 26747)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe0d090)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fcd14625700 (LWP 26748)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe0d390)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fcd13e24700 (LWP 26749)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe0d090)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fcd13623700 (LWP 26750)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d1f8, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f493b in virThreadPoolWorker (opaque=opaque@entry=0xe0d390)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fcd12e22700 (LWP 26751)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d1f8, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f493b in virThreadPoolWorker (opaque=opaque@entry=0xe0d090)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fcd12621700 (LWP 26752)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d1f8, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f493b in virThreadPoolWorker (opaque=opaque@entry=0xe0d390)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fcd11e20700 (LWP 26753)):
#0  0x00000030d0e0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000030d0e09ca6 in _L_lock_840 () from /lib64/libpthread.so.0
#2  0x00000030d0e09ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fcd162f416d in virMutexLock (m=<optimized out>)
    at util/threads-pthread.c:85
#4  0x00007fcd1632c651 in virDomainObjLock (obj=<optimized out>)
    at conf/domain_conf.c:14256
#5  0x00007fcd0daf05cc in qemuProcessHandleMonitorDestroy (mon=0x7fcccc0029e0, 
    vm=0x7fcccc00a850) at qemu/qemu_process.c:1026
#6  0x00007fcd0db01710 in qemuMonitorDispose (obj=0x7fcccc0029e0)
    at qemu/qemu_monitor.c:249
#7  0x00007fcd162fd4e3 in virObjectUnref (anyobj=<optimized out>)
    at util/virobject.c:139
#8  0x00007fcd0db027a9 in qemuMonitorClose (mon=<optimized out>)
    at qemu/qemu_monitor.c:860
#9  0x00007fcd0daf61ad in qemuProcessStop (driver=driver@entry=0x7fcd04079d50, 
    vm=vm@entry=0x7fcccc00a850, 
    reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED, flags=flags@entry=0)
    at qemu/qemu_process.c:4057
#10 0x00007fcd0db323cf in qemuDomainDestroyFlags (dom=<optimized out>, 
    flags=<optimized out>) at qemu/qemu_driver.c:1977
#11 0x00007fcd1637ff51 in virDomainDestroyFlags (
    domain=domain@entry=0x7fccf00c1830, flags=1) at libvirt.c:2256
#12 0x0000000000415dab in remoteDispatchDomainDestroyFlags (
    args=0x7fccf00c1870, rerr=0x7fcd11e1fc60, client=<optimized out>, 
    server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1329
#13 remoteDispatchDomainDestroyFlagsHelper (server=<optimized out>, 
    client=<optimized out>, msg=<optimized out>, rerr=0x7fcd11e1fc60, 
    args=0x7fccf00c1870, ret=<optimized out>) at remote_dispatch.h:1307
#14 0x00007fcd163eaba8 in virNetServerProgramDispatchCall (msg=0xe46100, 
    client=0xe45910, server=0xe1bda0, prog=0xe3fa40)
    at rpc/virnetserverprogram.c:431
#15 virNetServerProgramDispatch (prog=0xe3fa40, server=server@entry=0xe1bda0, 
    client=0xe45910, msg=0xe46100) at rpc/virnetserverprogram.c:304
#16 0x00007fcd163e5e41 in virNetServerProcessMsg (msg=<optimized out>, 
    prog=<optimized out>, client=<optimized out>, srv=0xe1bda0)
    at rpc/virnetserver.c:170
#17 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0xe1bda0)
    at rpc/virnetserver.c:191
#18 0x00007fcd162f487e in virThreadPoolWorker (opaque=opaque@entry=0xe0d090)
    at util/threadpool.c:144
#19 0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#20 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#21 0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fcd1161f700 (LWP 26755)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d1f8, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f493b in virThreadPoolWorker (opaque=opaque@entry=0xe0d390)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fcd0c463700 (LWP 3613)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe466d0)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fccdbfff700 (LWP 3614)):
#0  0x00000030d0e0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000030d0e09ca6 in _L_lock_840 () from /lib64/libpthread.so.0
#2  0x00000030d0e09ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fcd162f416d in virMutexLock (m=<optimized out>)
    at util/threads-pthread.c:85
#4  0x00007fcd0daee44d in qemuDriverLock (driver=<optimized out>)
    at qemu/qemu_conf.c:67
#5  0x00007fcd0db32341 in qemuDomainDestroyFlags (dom=0x7fccd40cdbf0, flags=1)
    at qemu/qemu_driver.c:1932
#6  0x00007fcd1637ff51 in virDomainDestroyFlags (
    domain=domain@entry=0x7fccd40cdbf0, flags=1) at libvirt.c:2256
#7  0x0000000000415dab in remoteDispatchDomainDestroyFlags (
    args=0x7fccd40cdc30, rerr=0x7fccdbffec60, client=<optimized out>, 
    server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:1329
#8  remoteDispatchDomainDestroyFlagsHelper (server=<optimized out>, 
    client=<optimized out>, msg=<optimized out>, rerr=0x7fccdbffec60, 
    args=0x7fccd40cdc30, ret=<optimized out>) at remote_dispatch.h:1307
#9  0x00007fcd163eaba8 in virNetServerProgramDispatchCall (msg=0xe43650, 
    client=0xe4f6d0, server=0xe1bda0, prog=0xe3fa40)
    at rpc/virnetserverprogram.c:431
#10 virNetServerProgramDispatch (prog=0xe3fa40, server=server@entry=0xe1bda0, 
    client=0xe4f6d0, msg=0xe43650) at rpc/virnetserverprogram.c:304
#11 0x00007fcd163e5e41 in virNetServerProcessMsg (msg=<optimized out>, 
    prog=<optimized out>, client=<optimized out>, srv=0xe1bda0)
    at rpc/virnetserver.c:170
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0xe1bda0)
    at rpc/virnetserver.c:191
#13 0x00007fcd162f487e in virThreadPoolWorker (opaque=opaque@entry=0xe454e0)
    at util/threadpool.c:144
#14 0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#15 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#16 0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fccdb7fe700 (LWP 3615)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe45700)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fcd0bc11700 (LWP 3616)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe468d0)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fcd0b410700 (LWP 3617)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe46d90)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fcd0ac0f700 (LWP 3618)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe46560)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fcd0a40e700 (LWP 3619)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007fcd162f429a in virCondWait (c=c@entry=0xe1d160, m=m@entry=0xe1d138)
    at util/threads-pthread.c:117
#2  0x00007fcd162f491b in virThreadPoolWorker (opaque=opaque@entry=0xe46560)
    at util/threadpool.c:103
#3  0x00007fcd162f3f36 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fcd1624a840 (LWP 26731)):
#0  0x00000030d0e0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000030d0e09ca6 in _L_lock_840 () from /lib64/libpthread.so.0
#2  0x00000030d0e09ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fcd162f416d in virMutexLock (m=<optimized out>)
    at util/threads-pthread.c:85
#4  0x00007fcd0daee44d in qemuDriverLock (driver=<optimized out>)
    at qemu/qemu_conf.c:67
#5  0x00007fcd0daf51cb in qemuProcessHandleShutdown (mon=<optimized out>, 
    vm=0x7fccc40c5ff0) at qemu/qemu_process.c:682
#6  0x00007fcd0db035b2 in qemuMonitorEmitShutdown (mon=0x7fccc40c5400)
    at qemu/qemu_monitor.c:980
#7  0x00007fcd0db1309e in qemuMonitorJSONIOProcessEvent (obj=0xe15f50, 
    mon=0x7fccc40c5400) at qemu/qemu_monitor_json.c:131
#8  qemuMonitorJSONIOProcessLine (msg=0x0, line=<optimized out>, 
    mon=<optimized out>) at qemu/qemu_monitor_json.c:160
#9  qemuMonitorJSONIOProcess (mon=mon@entry=0x7fccc40c5400, 
    data=0xe46420 "{\"timestamp\": {\"seconds\": 1348088643, \"microseconds\": 3279}, \"event\": \"SHUTDOWN\"}\r\n{\"timestamp\": {\"seconds\": 1348088643, \"microseconds\": 4623}, \"event\": \"STOP\"}\r\n", len=162, msg=msg@entry=0x0)
    at qemu/qemu_monitor_json.c:204
#10 0x00007fcd0db0204f in qemuMonitorIOProcess (mon=0x7fccc40c5400)
    at qemu/qemu_monitor.c:357
#11 qemuMonitorIO (watch=watch@entry=421, fd=<optimized out>, events=0, 
    events@entry=1, opaque=opaque@entry=0x7fccc40c5400)
    at qemu/qemu_monitor.c:598
#12 0x00007fcd162e18a6 in virEventPollDispatchHandles (fds=<optimized out>, 
    nfds=<optimized out>) at util/event_poll.c:485
#13 virEventPollRunOnce () at util/event_poll.c:632
#14 0x00007fcd162dff55 in virEventRunDefaultImpl () at util/event.c:247
#15 0x00007fcd163e699d in virNetServerRun (srv=0xe1bda0)
    at rpc/virnetserver.c:751
#16 0x000000000040c2ea in main (argc=<optimized out>, argv=<optimized out>)
    at libvirtd.c:1338


I often run the libguestfs test suite on this machine, and the test
suite creates and shuts down hundreds of transient guests, sometimes
in parallel.
Comment 1 Richard W.M. Jones 2012-09-20 07:03:13 EDT
It turns out that a test job was running (and has
blocked):

 8406 ?        Ss     0:22 whenjobsd
25069 ?        S      0:00  \_ /bin/bash -c /tmp/whenjobs5fb33aaa6c7103130629d3a
13998 ?        S      0:00      \_ make check
14055 ?        S      0:00          \_ /bin/sh -c fail= failcom='exit 1'; \ for 
31524 ?        S      0:00              \_ /bin/sh -c fail= failcom='exit 1'; \ 
31525 ?        S      0:00                  \_ make check
31611 ?        S      0:00                      \_ make check-TESTS
31612 ?        S      0:00                          \_ /bin/sh -c failed=0; all=
31613 ?        S      0:00                              \_ /bin/bash - ../../run
31651 ?        Sl     5:23                                  \_ /tmp/whenjobs5fb3

There are no non-root qemu processes running on the machine,
and libvirtd has no children.
Comment 2 Richard W.M. Jones 2012-09-20 07:04:57 EDT
Killing the test job didn't fix libvirtd.
Comment 3 Richard W.M. Jones 2012-09-20 07:13:18 EDT
The test which failed is our old friend test-parallel-mount-local:

https://github.com/libguestfs/libguestfs/blob/aeb2f76a585afdca07691771ab4d3d50cab2cbd0/tests/mount-local/test-parallel-mount-local.c

This is designed to be an aggressive parallel test of
libguestfs + the mount-local API.  It creates 12 transient
guests, runs some tests against each, then they all exit
roughly simultaneously.
Comment 4 Daniel Berrange 2012-09-20 07:15:53 EDT
This is the interesting thread:

Thread 10 (Thread 0x7fcd11e20700 (LWP 26753)):
#0  0x00000030d0e0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000030d0e09ca6 in _L_lock_840 () from /lib64/libpthread.so.0
#2  0x00000030d0e09ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fcd162f416d in virMutexLock (m=<optimized out>)
    at util/threads-pthread.c:85
#4  0x00007fcd1632c651 in virDomainObjLock (obj=<optimized out>)
    at conf/domain_conf.c:14256
#5  0x00007fcd0daf05cc in qemuProcessHandleMonitorDestroy (mon=0x7fcccc0029e0, 
    vm=0x7fcccc00a850) at qemu/qemu_process.c:1026
#6  0x00007fcd0db01710 in qemuMonitorDispose (obj=0x7fcccc0029e0)
    at qemu/qemu_monitor.c:249
#7  0x00007fcd162fd4e3 in virObjectUnref (anyobj=<optimized out>)
    at util/virobject.c:139
#8  0x00007fcd0db027a9 in qemuMonitorClose (mon=<optimized out>)
    at qemu/qemu_monitor.c:860
#9  0x00007fcd0daf61ad in qemuProcessStop (driver=driver@entry=0x7fcd04079d50, 
    vm=vm@entry=0x7fcccc00a850, 
    reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED, flags=flags@entry=0)
    at qemu/qemu_process.c:4057
#10 0x00007fcd0db323cf in qemuDomainDestroyFlags (dom=<optimized out>, 
    flags=<optimized out>) at qemu/qemu_driver.c:1977
#11 0x00007fcd1637ff51 in virDomainDestroyFlags (
    domain=domain@entry=0x7fccf00c1830, flags=1) at libvirt.c:2256


At frame #10 we are holding the domain lock, we call into qemuProcessStop() to cleanup QEMU, which triggers the monitor to close, which invokes qemuProcessHandleMonitorDestroy() which tries to obtain the domain lock again. This is a non-recursive lock, hence hang.   The other threads then hang on waiting for this thread

This exact same class of problem is also affecting the LXC driver, so we'll need the same type of solution in both bugs

https://bugzilla.redhat.com/show_bug.cgi?id=856950
Comment 5 Daniel Berrange 2012-09-20 07:16:09 EDT
Opps wrong bug, that should have been:

https://bugzilla.redhat.com/show_bug.cgi?id=848119
Comment 6 Cole Robinson 2012-10-17 16:50:45 EDT
Sounds like it's fixed by:

commit 25f582e36a1c066b6c82303b5e4f18eec337a25b
Author: Daniel P. Berrange <berrange@redhat.com>
Date:   Wed Sep 26 15:54:58 2012 +0100

    Fix (rare) deadlock in QEMU monitor callbacks
    
    Some users report (very rarely) seeing a deadlock in the QEMU
    monitor callbacks
Comment 7 Cole Robinson 2012-10-27 16:21:54 EDT
*** Bug 856639 has been marked as a duplicate of this bug. ***
Comment 8 Fedora Update System 2012-10-27 18:31:24 EDT
libvirt-0.10.2.1-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/libvirt-0.10.2.1-1.fc18
Comment 9 Fedora Update System 2012-10-28 12:33:32 EDT
Package libvirt-0.10.2.1-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-0.10.2.1-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17097/libvirt-0.10.2.1-1.fc18
then log in and leave karma (feedback).
Comment 10 Fedora Update System 2012-10-30 21:20:32 EDT
libvirt-0.10.2.1-2.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/libvirt-0.10.2.1-2.fc18
Comment 11 Cole Robinson 2012-11-28 09:56:31 EST
libvirt-0.10.2.1-2.fc18 now in stable

Note You need to log in before you can comment on or make changes to this bug.