Bug 1057535
| Summary: | libvirt hangs running 'virsh capabilities' | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] Virtualization Tools | Reporter: | Richard W.M. Jones <rjones> | ||||
| Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | |||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | unspecified | CC: | berrange, crobinso, rbalakri | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-04-10 17:36:12 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 910269 | ||||||
| Attachments: |
|
||||||
21825 ? Sl 5:32 /usr/sbin/libvirtd --timeout=30 PID 21825 (libvirtd) is unkillable. It is blocked on a futex: $ strace -p 21825 Process 21825 attached futex(0x7f91806bb664, FUTEX_WAIT_PRIVATE, 1, NULL [hangs here] $ ps -efL | grep libvirtd rjones 21825 1 21825 0 7 Jan16 ? 00:00:58 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 21826 0 7 Jan16 ? 00:00:30 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 21830 0 7 Jan16 ? 00:00:28 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 4519 0 7 Jan16 ? 00:00:30 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 30897 0 7 Jan20 ? 00:00:14 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 464 0 7 Jan20 ? 00:00:13 /usr/sbin/libvirtd --timeout=30 rjones 21825 1 467 0 7 Jan20 ? 00:00:13 /usr/sbin/libvirtd --timeout=30 Stack trace of libvirtd (there are no clients connected):
(gdb) t a a bt
Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0 0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2 0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3 0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a871b in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c62167e in qemuDomainCreateXML ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7 0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#9 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 6 (Thread 0x7f9174850700 (LWP 21830)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 5 (Thread 0x7f916b177700 (LWP 4519)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 4 (Thread 0x7f916a175700 (LWP 30897)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 3 (Thread 0x7f9169974700 (LWP 464)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 2 (Thread 0x7f912bfff700 (LWP 467)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0
#2 0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0
#3 0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0
#4 0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0
#5 0x00007f917f75db1f in main ()
Stack trace with one client hanging in virsh capabilities:
(gdb) t a a bt
Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0 0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2 0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3 0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a871b in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c62167e in qemuDomainCreateXML ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7 0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#9 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 6 (Thread 0x7f9174850700 (LWP 21830)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9174850700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 5 (Thread 0x7f916b177700 (LWP 4519)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916b177700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 4 (Thread 0x7f916a175700 (LWP 30897)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f916a175700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 3 (Thread 0x7f9169974700 (LWP 464)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f9169974700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 2 (Thread 0x7f912bfff700 (LWP 467)):
#0 __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f917bf00179 in _L_lock_814 () from /lib64/libpthread.so.0
#2 0x00007f917bf00048 in __GI___pthread_mutex_lock (mutex=0x7f91642f6540)
at ../nptl/pthread_mutex_lock.c:79
#3 0x00007f916c5a8613 in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a8bfa in virQEMUCapsInit ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c5d925b in virQEMUDriverCreateCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f916c5d94c3 in virQEMUDriverGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#7 0x00007f916c621b20 in qemuConnectGetCapabilities ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8 0x00007f917ed2efae in virConnectGetCapabilities () from /lib64/libvirt.so.0
#9 0x00007f917f77e284 in remoteDispatchConnectGetCapabilitiesHelper ()
#10 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#11 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#12 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#13 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#14 0x00007f917befdf33 in start_thread (arg=0x7f912bfff700)
at pthread_create.c:309
#15 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 1 (Thread 0x7f917f6fa880 (LWP 21825)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f917ec9ef56 in virCondWait () from /lib64/libvirt.so.0
#2 0x00007f917ec9f4d0 in virThreadPoolFree () from /lib64/libvirt.so.0
#3 0x00007f917ed862e2 in virNetServerDispose () from /lib64/libvirt.so.0
#4 0x00007f917ec8e41b in virObjectUnref () from /lib64/libvirt.so.0
#5 0x00007f917f75db1f in main ()
Hmm, that stack trace actually shows 5 separate calls to virsh capabilities, all being blocked by a single call to virDomainCreateXML, which is stuck waiting for a QEMU binary to respond to an attempt to query capabilities:
Thread 7 (Thread 0x7f9176053700 (LWP 21826)):
#0 0x00007f917b81aa8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f917ec623ca in virCommandProcessIO () from /lib64/libvirt.so.0
#2 0x00007f917ec66472 in virCommandRun () from /lib64/libvirt.so.0
#3 0x00007f916c5a6c90 in virQEMUCapsNewForBinary ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4 0x00007f916c5a871b in virQEMUCapsCacheLookup ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5 0x00007f916c62167e in qemuDomainCreateXML ()
from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#6 0x00007f917ed20d07 in virDomainCreateXML () from /lib64/libvirt.so.0
#7 0x00007f917f77deaf in remoteDispatchDomainCreateXMLHelper ()
#8 0x00007f917ed8c23a in virNetServerProgramDispatch ()
from /lib64/libvirt.so.0
#9 0x00007f917ed86df8 in virNetServerHandleJob () from /lib64/libvirt.so.0
#10 0x00007f917ec9f375 in virThreadPoolWorker () from /lib64/libvirt.so.0
#11 0x00007f917ec9ed7e in virThreadHelper () from /lib64/libvirt.so.0
#12 0x00007f917befdf33 in start_thread (arg=0x7f9176053700)
at pthread_create.c:309
#13 0x00007f917b824ead in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Is there a hung QEMU process on the machine that you can see that would correspond to this command invocation ?
I've just killed (with -9) the libvirtd instance. However I suspect that I know the answer. I was testing out strace of qemu this morning (see: https://www.redhat.com/archives/libguestfs/2014-January/msg00224.html ) which might have caused qemu to hang. Yeah, strace is certainly a likely candidate for messing things up when it re-parents and generally screws around with the process. FWIW, there's a trick to let you attach GDB to a QEMU process without using a wrapper script. https://www.berrange.com/posts/2011/10/12/debugging-early-startup-of-kvm-with-gdb-when-launched-by-libvirtd/ Sounds like this is NOTABUG |
Created attachment 854872 [details] LIBVIRT_DEBUG=1 virsh capabilities Description of problem: $ virsh capabilities hangs, with no further output. LIBVIRT_DEBUG=1 output is attached. Version-Release number of selected component (if applicable): libvirt-client-1.1.3.2-1.fc20.x86_64 libvirt-daemon-1.1.3.2-1.fc20.x86_64 How reproducible: Unknown Steps to Reproduce: 1. Run 'virsh capabilities' Additional info: This is *non root*. This also stopped libguestfs from working since the first thing it does is to query libvirt capabilities.