Bug 822971 - deadlock when unregistering event callbacks
Summary: deadlock when unregistering event callbacks
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Eric Blake
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-05-18 16:13 UTC by Zeeshan Ali
Modified: 2016-09-20 01:41 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-06-29 02:00:09 UTC
Embargoed:


Attachments (Terms of Use)

Description Zeeshan Ali 2012-05-18 16:13:59 UTC
Description of problem:

I got session libvirtd hung after I tried to managed-save a Windows XP domain. Here is the bt:

Thread 11 (Thread 0x7fcbb4e45700 (LWP 23186)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb4e45700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7fcbb4644700 (LWP 23187)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb4644700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7fcbb3e43700 (LWP 23188)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb3e43700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7fcbb3642700 (LWP 23189)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb3642700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7fcbb2e41700 (LWP 23190)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb2e41700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7fcbb2640700 (LWP 23191)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb2640700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7fcbb1e3f700 (LWP 23192)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb1e3f700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7fcbb163e700 (LWP 23193)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb163e700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7fcbb0e3d700 (LWP 23194)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb0e3d700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fcbb063c700 (LWP 23195)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb063c700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fcbb526a840 (LWP 23185)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000030bfa09f97 in _L_lock_863 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x00000030bfa09deb in __pthread_mutex_lock (mutex=0x7fcba8066628) at pthread_mutex_lock.c:65
#3  0x00007fcbb69ce0d0 in virDomainEventStateLock (state=0x7fcba8066610) at conf/domain_event.c:598
#4  virDomainEventStateDeregisterConn (conn=0x7fcba4000ca0, state=0x7fcba8066610) at conf/domain_event.c:1625
#5  0x0000000000464afc in qemudClose (conn=0x7fcba4000ca0) at qemu/qemu_driver.c:951
#6  0x00007fcbb69f4c4b in virReleaseConnect (conn=0x7fcba4000ca0) at datatypes.c:114
#7  0x00007fcbb69f4fa8 in virUnrefConnect (conn=0x7fcba4000ca0) at datatypes.c:149
#8  0x00007fcbb69cc487 in virDomainEventCallbackListPurgeMarked (cbList=0x7fcba8062ba0) at conf/domain_event.c:351
#9  virDomainEventStateFlush (state=0x7fcba8066610) at conf/domain_event.c:1422
#10 virDomainEventTimer (timer=<optimized out>, opaque=0x7fcba8066610) at conf/domain_event.c:637
#11 0x00007fcbb6976348 in virEventPollDispatchTimeouts () at util/event_poll.c:440
#12 virEventPollRunOnce () at util/event_poll.c:633
#13 0x00007fcbb6974c55 in virEventRunDefaultImpl () at util/event.c:247
#14 0x00007fcbb6a5813d in virNetServerRun (srv=0x1b9fbd0) at rpc/virnetserver.c:713
#15 0x0000000000420ddc in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1133

Version-Release number of selected component (if applicable): git master (commit f300c194fdfbd519013563dce5d410e32d7c24b8)

How reproducible: sometimes

Steps to Reproduce: virsh managedsave DOMAIN_NAME
  
Actual results: domain is saved but libvirtd hangs

Expected results: domain is saved and libvirt continues to run..

Comment 1 Daniel Berrangé 2012-05-21 11:20:16 UTC
As can be seen from the stack trace this is not actually anything todo with managed save. It is a simple deadlock in the event handling code

https://www.redhat.com/archives/libvir-list/2012-May/msg00995.html

Comment 2 Zeeshan Ali 2013-06-24 23:14:52 UTC
AFAICT, this is fixed. Since I filed this, I'll mark this directly as "VERIFIED"


Note You need to log in before you can comment on or make changes to this bug.