Description of problem: I got session libvirtd hung after I tried to managed-save a Windows XP domain. Here is the bt: Thread 11 (Thread 0x7fcbb4e45700 (LWP 23186)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb4e45700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 10 (Thread 0x7fcbb4644700 (LWP 23187)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb4644700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 9 (Thread 0x7fcbb3e43700 (LWP 23188)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb3e43700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 8 (Thread 0x7fcbb3642700 (LWP 23189)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb3642700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 7 (Thread 0x7fcbb2e41700 (LWP 23190)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb2e41700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 6 (Thread 0x7fcbb2640700 (LWP 23191)): ---Type <return> to continue, or q <return> to quit--- #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb2640700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 5 (Thread 0x7fcbb1e3f700 (LWP 23192)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb1e3f700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 4 (Thread 0x7fcbb163e700 (LWP 23193)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb163e700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 3 (Thread 0x7fcbb0e3d700 (LWP 23194)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb0e3d700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 2 (Thread 0x7fcbb063c700 (LWP 23195)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165 #1 0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117 #2 0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103 #3 0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030bfa07d90 in start_thread (arg=0x7fcbb063c700) at pthread_create.c:309 #5 0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 1 (Thread 0x7fcbb526a840 (LWP 23185)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136 #1 0x00000030bfa09f97 in _L_lock_863 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #2 0x00000030bfa09deb in __pthread_mutex_lock (mutex=0x7fcba8066628) at pthread_mutex_lock.c:65 #3 0x00007fcbb69ce0d0 in virDomainEventStateLock (state=0x7fcba8066610) at conf/domain_event.c:598 #4 virDomainEventStateDeregisterConn (conn=0x7fcba4000ca0, state=0x7fcba8066610) at conf/domain_event.c:1625 #5 0x0000000000464afc in qemudClose (conn=0x7fcba4000ca0) at qemu/qemu_driver.c:951 #6 0x00007fcbb69f4c4b in virReleaseConnect (conn=0x7fcba4000ca0) at datatypes.c:114 #7 0x00007fcbb69f4fa8 in virUnrefConnect (conn=0x7fcba4000ca0) at datatypes.c:149 #8 0x00007fcbb69cc487 in virDomainEventCallbackListPurgeMarked (cbList=0x7fcba8062ba0) at conf/domain_event.c:351 #9 virDomainEventStateFlush (state=0x7fcba8066610) at conf/domain_event.c:1422 #10 virDomainEventTimer (timer=<optimized out>, opaque=0x7fcba8066610) at conf/domain_event.c:637 #11 0x00007fcbb6976348 in virEventPollDispatchTimeouts () at util/event_poll.c:440 #12 virEventPollRunOnce () at util/event_poll.c:633 #13 0x00007fcbb6974c55 in virEventRunDefaultImpl () at util/event.c:247 #14 0x00007fcbb6a5813d in virNetServerRun (srv=0x1b9fbd0) at rpc/virnetserver.c:713 #15 0x0000000000420ddc in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1133 Version-Release number of selected component (if applicable): git master (commit f300c194fdfbd519013563dce5d410e32d7c24b8) How reproducible: sometimes Steps to Reproduce: virsh managedsave DOMAIN_NAME Actual results: domain is saved but libvirtd hangs Expected results: domain is saved and libvirt continues to run..
As can be seen from the stack trace this is not actually anything todo with managed save. It is a simple deadlock in the event handling code https://www.redhat.com/archives/libvir-list/2012-May/msg00995.html
AFAICT, this is fixed. Since I filed this, I'll mark this directly as "VERIFIED"