Bug 822971 - deadlock when unregistering event callbacks
deadlock when unregistering event callbacks
Status: CLOSED CURRENTRELEASE
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
unspecified
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Eric Blake
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-18 12:13 EDT by Zeeshan Ali
Modified: 2016-09-19 21:41 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-28 22:00:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Zeeshan Ali 2012-05-18 12:13:59 EDT
Description of problem:

I got session libvirtd hung after I tried to managed-save a Windows XP domain. Here is the bt:

Thread 11 (Thread 0x7fcbb4e45700 (LWP 23186)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb4e45700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7fcbb4644700 (LWP 23187)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb4644700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7fcbb3e43700 (LWP 23188)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb3e43700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7fcbb3642700 (LWP 23189)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb3642700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 7 (Thread 0x7fcbb2e41700 (LWP 23190)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb2e41700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7fcbb2640700 (LWP 23191)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb2640700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7fcbb1e3f700 (LWP 23192)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb1e3f700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7fcbb163e700 (LWP 23193)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb163e700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7fcbb0e3d700 (LWP 23194)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb0e3d700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fcbb063c700 (LWP 23195)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:165
#1  0x00007fcbb69888da in virCondWait (c=<optimized out>, m=<optimized out>) at util/threads-pthread.c:117
#2  0x00007fcbb6988f6b in virThreadPoolWorker (opaque=<optimized out>) at util/threadpool.c:103
#3  0x00007fcbb6988576 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161
#4  0x00000030bfa07d90 in start_thread (arg=0x7fcbb063c700) at pthread_create.c:309
#5  0x00000030bf6f0f5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fcbb526a840 (LWP 23185)):
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00000030bfa09f97 in _L_lock_863 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x00000030bfa09deb in __pthread_mutex_lock (mutex=0x7fcba8066628) at pthread_mutex_lock.c:65
#3  0x00007fcbb69ce0d0 in virDomainEventStateLock (state=0x7fcba8066610) at conf/domain_event.c:598
#4  virDomainEventStateDeregisterConn (conn=0x7fcba4000ca0, state=0x7fcba8066610) at conf/domain_event.c:1625
#5  0x0000000000464afc in qemudClose (conn=0x7fcba4000ca0) at qemu/qemu_driver.c:951
#6  0x00007fcbb69f4c4b in virReleaseConnect (conn=0x7fcba4000ca0) at datatypes.c:114
#7  0x00007fcbb69f4fa8 in virUnrefConnect (conn=0x7fcba4000ca0) at datatypes.c:149
#8  0x00007fcbb69cc487 in virDomainEventCallbackListPurgeMarked (cbList=0x7fcba8062ba0) at conf/domain_event.c:351
#9  virDomainEventStateFlush (state=0x7fcba8066610) at conf/domain_event.c:1422
#10 virDomainEventTimer (timer=<optimized out>, opaque=0x7fcba8066610) at conf/domain_event.c:637
#11 0x00007fcbb6976348 in virEventPollDispatchTimeouts () at util/event_poll.c:440
#12 virEventPollRunOnce () at util/event_poll.c:633
#13 0x00007fcbb6974c55 in virEventRunDefaultImpl () at util/event.c:247
#14 0x00007fcbb6a5813d in virNetServerRun (srv=0x1b9fbd0) at rpc/virnetserver.c:713
#15 0x0000000000420ddc in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1133

Version-Release number of selected component (if applicable): git master (commit f300c194fdfbd519013563dce5d410e32d7c24b8)

How reproducible: sometimes

Steps to Reproduce: virsh managedsave DOMAIN_NAME
  
Actual results: domain is saved but libvirtd hangs

Expected results: domain is saved and libvirt continues to run..
Comment 1 Daniel Berrange 2012-05-21 07:20:16 EDT
As can be seen from the stack trace this is not actually anything todo with managed save. It is a simple deadlock in the event handling code

https://www.redhat.com/archives/libvir-list/2012-May/msg00995.html
Comment 2 Zeeshan Ali 2013-06-24 19:14:52 EDT
AFAICT, this is fixed. Since I filed this, I'll mark this directly as "VERIFIED"

Note You need to log in before you can comment on or make changes to this bug.