Bug 811497

Summary: Deadlock in qemu driver on forced console connection
Product: Red Hat Enterprise Linux 6 Reporter: Peter Krempa <pkrempa>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3CC: acathrow, dallan, dyasny, dyuan, eblake, gsun, mzhan, rwu, veillard, whuang, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
No Documentation needed (bug in new feature)
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 06:53:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Leaves the console connection in a state, where the next forced connection causes a deadlock. none

Description Peter Krempa 2012-04-11 09:26:32 UTC
Created attachment 576725 [details]
Leaves the console connection in a state, where the next forced connection causes a deadlock.

Description of problem:
The qemu driver deadlocks, when a newly open console connection cleans up a already dead connection

Version-Release number of selected component (if applicable):
libvirt-0.9.10-4.el6 and later

How reproducible:
100%

Steps to Reproduce:
1. Start a guest with a serial console (the guest OS doesn't need to support it)
2. modify guest name in console_deadlock.py reproducer script
3. run console_deadlock.py
4. run 'virsh console <guestname> --force'
  
Actual results:
Virsh locks up and the daemon deadlocks:

Thread 11 (Thread 0x7f4caa6d4700 (LWP 22715)):
#0  0x00007f4cb15d3f14 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f4cb15cf411 in _L_lock_500 () from /lib64/libpthread.so.0
#2  0x00007f4cb15cf22a in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000462bbd in qemudClose ()
#4  0x00007f4cb29438eb in virReleaseConnect () from /usr/lib64/libvirt.so.0
#5  0x00007f4cb2945c8c in virUnrefStream () from /usr/lib64/libvirt.so.0
#6  0x00007f4cb29691de in virStreamFree () from /usr/lib64/libvirt.so.0
#7  0x00007f4cb2935a5d in virConsoleHashEntryFree () from /usr/lib64/libvirt.so.0
#8  0x00007f4cb28e3282 in virHashRemoveEntry () from /usr/lib64/libvirt.so.0
#9  0x00007f4cb2935c4e in virConsoleOpen () from /usr/lib64/libvirt.so.0
#10 0x00000000004526e9 in qemuDomainOpenConsole ()
#11 0x00007f4cb296e1f1 in virDomainOpenConsole () from /usr/lib64/libvirt.so.0
#12 0x00000000004361e4 in remoteDispatchDomainOpenConsoleHelper ()
#13 0x00007f4cb29ac375 in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0
#14 0x00007f4cb29a6e11 in virNetServerHandleJob () from /usr/lib64/libvirt.so.0
#15 0x00007f4cb28d497d in virThreadPoolWorker () from /usr/lib64/libvirt.so.0
#16 0x00007f4cb28d3ff6 in virThreadHelper () from /usr/lib64/libvirt.so.0
#17 0x00007f4cb15ccc5c in start_thread () from /lib64/libpthread.so.0
#18 0x00007f4cb1313fcd in clone () from /lib64/libc.so.6

Comment 2 Peter Krempa 2012-04-11 09:31:26 UTC
Fixed upstream:

commit 3d3de46a6772baabb6c203a7961a790af6e8d08c
Author: Peter Krempa <pkrempa>
Date:   Tue Apr 10 11:56:21 2012 +0200

    qemu: Fix deadlock when qemuDomainOpenConsole cleans up a connection

Comment 6 Eric Blake 2012-04-18 17:23:48 UTC
Back to POST; 0.9.10-12.el6 mistakenly applied the patch twice instead of once, with the second application affecting the wrong context and completely breaking the binary.

Comment 7 zhe peng 2012-04-19 05:44:27 UTC
I can reproduce this issue with libvirt-0.9.10-4.el6.

verify with libvirt-0.9.10-13.el6.x86_64
step: 
1. Start a guest with a serial console (the guest OS doesn't need to support
it)
2. modify guest name in console_deadlock.py reproducer script
3. run console_deadlock.py
4. run 'virsh console <guestname> --force'

virsh not locks up,the daemon not deadlocks.

Comment 10 zhe peng 2012-04-20 02:28:42 UTC
according to comment 7 , move to verified.

Comment 11 Peter Krempa 2012-05-02 09:35:20 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No Documentation needed (bug in new feature)

Comment 13 errata-xmlrpc 2012-06-20 06:53:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html