Bug 848119

Summary: lxc connections are ending in deadlock if the connection is interrupted with a ^c
Product: [Fedora] Fedora Reporter: Daniel Walsh <dwalsh>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: berrange, clalancette, crobinso, dougsland, itamar, jforbes, jyang, laine, libvirt-maint, rjones, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-11-28 09:56:39 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Daniel Walsh 2012-08-14 12:36:29 EDT
virt-sandbox-service create -c -u httpd.service apache1
virt-sandbox-service start apache1

This will cause libvirt connections to hang.  You have to killall -9 libvirtd to get it running again.
Comment 1 Daniel Berrange 2012-08-14 12:41:07 EDT
This is a deadlock in the auto-destroy code, related to the monitor client

#0  0x00000038a0a0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038a0a09ca6 in _L_lock_840 () from /lib64/libpthread.so.0
#2  0x00000038a0a09ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f4bd9579d55 in virMutexLock (m=<optimized out>) at util/threads-pthread.c:85
#4  0x00007f4bcacc7597 in lxcDriverLock (driver=0x7f4bc40c8290) at lxc/lxc_conf.h:81
#5  virLXCProcessMonitorEOFNotify (mon=<optimized out>, vm=0x7f4bb4000b00) at lxc/lxc_process.c:581
#6  0x00007f4bd9645c91 in virNetClientCloseLocked (client=client@entry=0x7f4bb4009e60)
    at rpc/virnetclient.c:554
#7  0x00007f4bd96460f8 in virNetClientIOEventLoopPassTheBuck (thiscall=0x0, client=0x7f4bb4009e60)
    at rpc/virnetclient.c:1306
#8  virNetClientIOEventLoopPassTheBuck (client=0x7f4bb4009e60, thiscall=0x0)
    at rpc/virnetclient.c:1287
#9  0x00007f4bd96467a2 in virNetClientCloseInternal (reason=3, client=0x7f4bb4009e60)
    at rpc/virnetclient.c:589
#10 virNetClientCloseInternal (client=0x7f4bb4009e60, reason=3) at rpc/virnetclient.c:561
#11 0x00007f4bcacc4a82 in virLXCMonitorClose (mon=0x7f4bb4000a00) at lxc/lxc_monitor.c:201
#12 0x00007f4bcacc55ac in virLXCProcessCleanup (reason=<optimized out>, vm=0x7f4bb4000b00, 
    driver=0x7f4bc40c8290) at lxc/lxc_process.c:240
#13 virLXCProcessStop (driver=0x7f4bc40c8290, vm=vm@entry=0x7f4bb4000b00, 
    reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED) at lxc/lxc_process.c:735
#14 0x00007f4bcacc5bd2 in virLXCProcessAutoDestroyDom (payload=<optimized out>, 
    name=0x7f4bb4003c80, opaque=0x7fff41af2df0) at lxc/lxc_process.c:94
#15 0x00007f4bd9586649 in virHashForEach (table=0x7f4bc409b270, 
    iter=iter@entry=0x7f4bcacc5ab0 <virLXCProcessAutoDestroyDom>, data=data@entry=0x7fff41af2df0)
    at util/virhash.c:514
#16 0x00007f4bcacc52d7 in virLXCProcessAutoDestroyRun (driver=driver@entry=0x7f4bc40c8290, 
    conn=conn@entry=0x7f4bb8000ab0) at lxc/lxc_process.c:120
#17 0x00007f4bcacca628 in lxcClose (conn=0x7f4bb8000ab0) at lxc/lxc_driver.c:128
#18 0x00007f4bd95e67ab in virReleaseConnect (conn=conn@entry=0x7f4bb8000ab0) at datatypes.c:114

I'm not understanding by virDomainDestroy doesn't have the same issue, since it calls virLXCProcessStop with the driver lock held too.
Comment 2 Daniel Berrange 2012-09-20 07:16:56 EDT
We've now seen the same issue with virDomainDestroy in the QEMU driver

Comment 3 Cole Robinson 2012-10-17 16:47:41 EDT
I'm thinking this is fixed by:

commit 36c1fc189d0a04b3ea916dd3533dfa6f5bc3b8ba
Author: Daniel P. Berrange <berrange@redhat.com>
Date:   Wed Sep 26 15:08:20 2012 +0100

    Fix deadlock in handling EOF in LXC monitor
    Depending on the scenario in which LXC containers exit, it is
    possible for the EOF callback of the LXC monitor to deadlock
    the driver.
Comment 4 Daniel Walsh 2012-10-26 15:23:44 EDT
Any chance we can get a newer libvirt into F18.
Comment 5 Fedora Update System 2012-10-27 18:31:34 EDT
libvirt- has been submitted as an update for Fedora 18.
Comment 6 Fedora Update System 2012-10-28 12:33:46 EDT
Package libvirt-
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
Comment 7 Fedora Update System 2012-10-30 21:20:42 EDT
libvirt- has been submitted as an update for Fedora 18.
Comment 8 Cole Robinson 2012-11-28 09:56:39 EST
libvirt- now in stable