Bug 1057535 - libvirt hangs running 'virsh capabilities'
Summary: libvirt hangs running 'virsh capabilities'
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs
TreeView+ depends on / blocked
 
Reported: 2014-01-24 10:00 UTC by Richard W.M. Jones
Modified: 2016-04-10 17:36 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2016-04-10 17:36:12 UTC
Embargoed:

Attachments (Terms of Use)
LIBVIRT_DEBUG=1 virsh capabilities (38.96 KB, text/plain)
2014-01-24 10:00 UTC, Richard W.M. Jones 		no flags Details
View All

Description Richard W.M. Jones 2014-01-24 10:00:44 UTC 
Created attachment 854872 [details]
LIBVIRT_DEBUG=1 virsh capabilities

Description of problem:

$ virsh capabilities

hangs, with no further output.

LIBVIRT_DEBUG=1 output is attached.

Version-Release number of selected component (if applicable):

libvirt-client-1.1.3.2-1.fc20.x86_64
libvirt-daemon-1.1.3.2-1.fc20.x86_64

How reproducible:

Unknown

Steps to Reproduce:
1. Run 'virsh capabilities'

Additional info:

This is *non root*.

This also stopped libguestfs from working since the first thing
it does is to query libvirt capabilities.
Comment 1 Richard W.M. Jones 2014-01-24 10:01:03 UTC 
21825 ?        Sl     5:32 /usr/sbin/libvirtd --timeout=30
Comment 2 Richard W.M. Jones 2014-01-24 10:05:12 UTC 
PID 21825 (libvirtd) is unkillable.

It is blocked on a futex:

$ strace -p 21825
Process 21825 attached
futex(0x7f91806bb664, FUTEX_WAIT_PRIVATE, 1, NULL [hangs here]

$ ps -efL | grep libvirtd
rjones   21825     1 21825  0    7 Jan16 ?        00:00:58 /usr/sbin/libvirtd --timeout=30
rjones   21825     1 21826  0    7 Jan16 ?        00:00:30 /usr/sbin/libvirtd --timeout=30
rjones   21825     1 21830  0    7 Jan16 ?        00:00:28 /usr/sbin/libvirtd --timeout=30
rjones   21825     1  4519  0    7 Jan16 ?        00:00:30 /usr/sbin/libvirtd --timeout=30
rjones   21825     1 30897  0    7 Jan20 ?        00:00:14 /usr/sbin/libvirtd --timeout=30
rjones   21825     1   464  0    7 Jan20 ?        00:00:13 /usr/sbin/libvirtd --timeout=30
rjones   21825     1   467  0    7 Jan20 ?        00:00:13 /usr/sbin/libvirtd --timeout=30
Comment 3 Richard W.M. Jones 2014-01-24 10:09:42 UTC 
Stack trace of libvirtd (there are no clients connected):

(gdb) t a a bt

Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0  0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a871b in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c62167e in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7  0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8  0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#9  0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
    at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f9174850700 (LWP 21830)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f916b177700 (LWP 4519)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f916a175700 (LWP 30897)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f9169974700 (LWP 464)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f912bfff700 (LWP 467)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0
#3  0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0
#4  0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0
#5  0x00007f917f75db1f in main ()
Comment 4 Richard W.M. Jones 2014-01-24 10:16:58 UTC 
Stack trace with one client hanging in virsh capabilities:

(gdb) t a a bt

Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0  0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a871b in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c62167e in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7  0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8  0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#9  0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
    at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f9174850700 (LWP 21830)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f916b177700 (LWP 4519)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f916a175700 (LWP 30897)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f9169974700 (LWP 464)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f912bfff700 (LWP 467)):
#0  __lll_lock_wait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2  0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
    at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a8bfa in virQEMUCapsInit ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7  0x00007f916c621b20 in qemuConnectGetCapabilities ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9  0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700)
    at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)):
#0  pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0
#3  0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0
#4  0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0
#5  0x00007f917f75db1f in main ()
Comment 5 Daniel Berrangé 2014-01-24 10:28:23 UTC 
Hmm, that stack trace actually shows 5 separate calls to virsh capabilities, all being blocked by a single call to virDomainCreateXML, which is stuck waiting for a QEMU binary to respond to an attempt to query capabilities:


Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0  0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007f916c5a871b in virQEMUCapsCacheLookup ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007f916c62167e in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6  0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7  0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8  0x00007f917ed8c23a in virNetServerProgramDispatch ()
   from /lib64/libvirt.so.0
#9  0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
    at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Is there a hung QEMU process on the machine that you can see that would correspond to this command invocation ?
Comment 6 Richard W.M. Jones 2014-01-24 10:32:53 UTC 
I've just killed (with -9) the libvirtd instance.

However I suspect that I know the answer.  I was testing out
strace of qemu this morning (see:
https://www.redhat.com/archives/libguestfs/2014-January/msg00224.html )
which might have caused qemu to hang.
Comment 7 Daniel Berrangé 2014-01-24 10:36:56 UTC 
Yeah, strace is certainly a likely candidate for messing things up when it re-parents and generally screws around with the process.

FWIW, there's a trick to let you attach GDB to a QEMU process without using a wrapper script. 

https://www.berrange.com/posts/2011/10/12/debugging-early-startup-of-kvm-with-gdb-when-launched-by-libvirtd/
Comment 8 Cole Robinson 2016-04-10 17:36:12 UTC 
Sounds like this is NOTABUG
Note You need to log in before you can comment on or make changes to this bug.