Bug 848119

Summary:	lxc connections are ending in deadlock if the connection is interrupted with a ^c
Product:	[Fedora] Fedora	Reporter:	Daniel Walsh <dwalsh>
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	18	CC:	berrange, clalancette, crobinso, dougsland, itamar, jforbes, jyang, laine, libvirt-maint, rjones, veillard, virt-maint
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-11-28 14:56:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Daniel Walsh 2012-08-14 16:36:29 UTC

virt-sandbox-service create -c -u httpd.service apache1
virt-sandbox-service start apache1
^c

This will cause libvirt connections to hang.  You have to killall -9 libvirtd to get it running again.

Comment 1 Daniel Berrangé 2012-08-14 16:41:07 UTC

This is a deadlock in the auto-destroy code, related to the monitor client

#0  0x00000038a0a0de4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038a0a09ca6 in _L_lock_840 () from /lib64/libpthread.so.0
#2  0x00000038a0a09ba8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f4bd9579d55 in virMutexLock (m=<optimized out>) at util/threads-pthread.c:85
#4  0x00007f4bcacc7597 in lxcDriverLock (driver=0x7f4bc40c8290) at lxc/lxc_conf.h:81
#5  virLXCProcessMonitorEOFNotify (mon=<optimized out>, vm=0x7f4bb4000b00) at lxc/lxc_process.c:581
#6  0x00007f4bd9645c91 in virNetClientCloseLocked (client=client@entry=0x7f4bb4009e60)
    at rpc/virnetclient.c:554
#7  0x00007f4bd96460f8 in virNetClientIOEventLoopPassTheBuck (thiscall=0x0, client=0x7f4bb4009e60)
    at rpc/virnetclient.c:1306
#8  virNetClientIOEventLoopPassTheBuck (client=0x7f4bb4009e60, thiscall=0x0)
    at rpc/virnetclient.c:1287
#9  0x00007f4bd96467a2 in virNetClientCloseInternal (reason=3, client=0x7f4bb4009e60)
    at rpc/virnetclient.c:589
#10 virNetClientCloseInternal (client=0x7f4bb4009e60, reason=3) at rpc/virnetclient.c:561
#11 0x00007f4bcacc4a82 in virLXCMonitorClose (mon=0x7f4bb4000a00) at lxc/lxc_monitor.c:201
#12 0x00007f4bcacc55ac in virLXCProcessCleanup (reason=<optimized out>, vm=0x7f4bb4000b00, 
    driver=0x7f4bc40c8290) at lxc/lxc_process.c:240
#13 virLXCProcessStop (driver=0x7f4bc40c8290, vm=vm@entry=0x7f4bb4000b00, 
    reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED) at lxc/lxc_process.c:735
#14 0x00007f4bcacc5bd2 in virLXCProcessAutoDestroyDom (payload=<optimized out>, 
    name=0x7f4bb4003c80, opaque=0x7fff41af2df0) at lxc/lxc_process.c:94
#15 0x00007f4bd9586649 in virHashForEach (table=0x7f4bc409b270, 
    iter=iter@entry=0x7f4bcacc5ab0 <virLXCProcessAutoDestroyDom>, data=data@entry=0x7fff41af2df0)
    at util/virhash.c:514
#16 0x00007f4bcacc52d7 in virLXCProcessAutoDestroyRun (driver=driver@entry=0x7f4bc40c8290, 
    conn=conn@entry=0x7f4bb8000ab0) at lxc/lxc_process.c:120
#17 0x00007f4bcacca628 in lxcClose (conn=0x7f4bb8000ab0) at lxc/lxc_driver.c:128
#18 0x00007f4bd95e67ab in virReleaseConnect (conn=conn@entry=0x7f4bb8000ab0) at datatypes.c:114

I'm not understanding by virDomainDestroy doesn't have the same issue, since it calls virLXCProcessStop with the driver lock held too.

Comment 2 Daniel Berrangé 2012-09-20 11:16:56 UTC

We've now seen the same issue with virDomainDestroy in the QEMU driver

https://bugzilla.redhat.com/show_bug.cgi?id=859009

Comment 3 Cole Robinson 2012-10-17 20:47:41 UTC

I'm thinking this is fixed by:

commit 36c1fc189d0a04b3ea916dd3533dfa6f5bc3b8ba
Author: Daniel P. Berrange <berrange>
Date:   Wed Sep 26 15:08:20 2012 +0100

    Fix deadlock in handling EOF in LXC monitor
    
    Depending on the scenario in which LXC containers exit, it is
    possible for the EOF callback of the LXC monitor to deadlock
    the driver.

Comment 4 Daniel Walsh 2012-10-26 19:23:44 UTC

Any chance we can get a newer libvirt into F18.

Comment 5 Fedora Update System 2012-10-27 22:31:34 UTC

libvirt-0.10.2.1-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/libvirt-0.10.2.1-1.fc18

Comment 6 Fedora Update System 2012-10-28 16:33:46 UTC

Package libvirt-0.10.2.1-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-0.10.2.1-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17097/libvirt-0.10.2.1-1.fc18
then log in and leave karma (feedback).

Comment 7 Fedora Update System 2012-10-31 01:20:42 UTC

libvirt-0.10.2.1-2.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/libvirt-0.10.2.1-2.fc18

Comment 8 Cole Robinson 2012-11-28 14:56:39 UTC

libvirt-0.10.2.1-2.fc18 now in stable