RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 995795 - Deadlock on libvirt when playing with hotplug and add/remove vm.
Summary: Deadlock on libvirt when playing with hotplug and add/remove vm.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Martin Kletzander
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-11 07:49 UTC by chhu
Modified: 2016-04-26 14:10 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-26 13:44:30 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description chhu 2013-08-11 07:49:20 UTC 
Description of problem:
Deadlock on libvirt when playing with hotplug and add/remove vm.

Version-Release number of selected component (if applicable):
libvirt-0.10.2-21.el6.x86_64
qemu-kvm-0.12.1.2-2.385.el6.x86_64
2.6.32-396.el6.x86_64

How reproducible:
80%


1. Prepare and start a guest.

2. Open the first terminal and run the following commands(add your domain name and image name):
# while true; do virsh attach-disk <domain> /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb; sleep 2; done

3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images' name from r1 to r10, and with guests' name from r1 to r10. Open the second terminal and run the following commands:

# while true; do for i in {1..10}; do virsh create /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy r$i; done; done

4. Wait for more than 10 hours

5. Deadlock on libvirt
# virsh list --all
error: Failed to reconnect to the hypervisor
error: no valid connection
error: Cannot recv data: Connection reset by peer

6. Check the libvirtd service
# service libvirtd status
libvirtd (pid  6757) is running...


Actual results:
Met deadlock, "Failed to reconnect to the hypervisor".

Expected results:
libvirt work well
Comment 2 Martin Kletzander 2013-08-13 12:16:58 UTC 
What is the state libvirtd ends up in?  Please provide a backtrace of the process obtained by issuing a 't a a bt full' command in attached gdb.  Also provide a debug log from the same time when the backtrace was extracted.  Thank you.
Comment 6 chhu 2013-09-12 08:04:49 UTC 
(In reply to chhu from comment #0)
> Description of problem:
> Deadlock on libvirt when playing with hotplug and add/remove vm.
> 
> Version-Release number of selected component (if applicable):
> libvirt-0.10.2-21.el6.x86_64
> qemu-kvm-0.12.1.2-2.385.el6.x86_64
> 2.6.32-396.el6.x86_64
> 
> How reproducible:
> 80%
> 
> 
> 1. Prepare and start a guest.
> 
> 2. Open the first terminal and run the following commands(add your domain
> name and image name):
> # while true; do virsh attach-disk <domain>
> /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb;
> sleep 2; done
> 
> 3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images'
> name from r1 to r10, and with guests' name from r1 to r10. Open the second
> terminal and run the following commands:
> 
> # while true; do for i in {1..10}; do virsh create
> /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy
> r$i; done; done
> 
> 4. Wait for more than 10 hours
> 
> 5. Deadlock on libvirt
> # virsh list --all
> error: Failed to reconnect to the hypervisor
> error: no valid connection
> error: Cannot recv data: Connection reset by peer
> 
> 6. Check the libvirtd service
> # service libvirtd status
> libvirtd (pid  6757) is running...
> 
> 
> Actual results:
> Met deadlock, "Failed to reconnect to the hypervisor".
> 
> Expected results:
> libvirt work well


Rerun the test with packages:
libvirt-0.10.2-24.el6.x86_64
qemu-kvm-0.12.1.2-2.400.el6.x86_64
kernel:2.6.32-416.el6.x86_64

The test last for over 48 hours, and don't hit this issue.
Comment 7 chhu 2013-09-13 08:16:13 UTC 
(In reply to chhu from comment #6)
> (In reply to chhu from comment #0)
> > Description of problem:
> > Deadlock on libvirt when playing with hotplug and add/remove vm.
> > 
> > Version-Release number of selected component (if applicable):
> > libvirt-0.10.2-21.el6.x86_64
> > qemu-kvm-0.12.1.2-2.385.el6.x86_64
> > 2.6.32-396.el6.x86_64
> > 
> > How reproducible:
> > 80%
> > 
> > 
> > 1. Prepare and start a guest.
> > 
> > 2. Open the first terminal and run the following commands(add your domain
> > name and image name):
> > # while true; do virsh attach-disk <domain>
> > /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb;
> > sleep 2; done
> > 
> > 3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images'
> > name from r1 to r10, and with guests' name from r1 to r10. Open the second
> > terminal and run the following commands:
> > 
> > # while true; do for i in {1..10}; do virsh create
> > /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy
> > r$i; done; done
> > 
> > 4. Wait for more than 10 hours
> > 
> > 5. Deadlock on libvirt
> > # virsh list --all
> > error: Failed to reconnect to the hypervisor
> > error: no valid connection
> > error: Cannot recv data: Connection reset by peer
> > 
> > 6. Check the libvirtd service
> > # service libvirtd status
> > libvirtd (pid  6757) is running...
> > 
> > 
> > Actual results:
> > Met deadlock, "Failed to reconnect to the hypervisor".
> > 
> > Expected results:
> > libvirt work well
> 
> 
> Rerun the test with packages:
> libvirt-0.10.2-24.el6.x86_64
> qemu-kvm-0.12.1.2-2.400.el6.x86_64
> kernel:2.6.32-416.el6.x86_64
> 
> The test last for over 48 hours, and don't hit this issue.

The test last for over 72 hours, and have not hit this issue.
Comment 8 chhu 2013-09-16 03:11:38 UTC 
(In reply to chhu from comment #7)
> (In reply to chhu from comment #6)
> > (In reply to chhu from comment #0)
> > > Description of problem:
> > > Deadlock on libvirt when playing with hotplug and add/remove vm.
> > > 
> > > Version-Release number of selected component (if applicable):
> > > libvirt-0.10.2-21.el6.x86_64
> > > qemu-kvm-0.12.1.2-2.385.el6.x86_64
> > > 2.6.32-396.el6.x86_64
> > > 
> > > How reproducible:
> > > 80%
> > > 
> > > 
> > > 1. Prepare and start a guest.
> > > 
> > > 2. Open the first terminal and run the following commands(add your domain
> > > name and image name):
> > > # while true; do virsh attach-disk <domain>
> > > /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb;
> > > sleep 2; done
> > > 
> > > 3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images'
> > > name from r1 to r10, and with guests' name from r1 to r10. Open the second
> > > terminal and run the following commands:
> > > 
> > > # while true; do for i in {1..10}; do virsh create
> > > /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy
> > > r$i; done; done
> > > 
> > > 4. Wait for more than 10 hours
> > > 
> > > 5. Deadlock on libvirt
> > > # virsh list --all
> > > error: Failed to reconnect to the hypervisor
> > > error: no valid connection
> > > error: Cannot recv data: Connection reset by peer
> > > 
> > > 6. Check the libvirtd service
> > > # service libvirtd status
> > > libvirtd (pid  6757) is running...
> > > 
> > > 
> > > Actual results:
> > > Met deadlock, "Failed to reconnect to the hypervisor".
> > > 
> > > Expected results:
> > > libvirt work well
> > 
> > 
> > Rerun the test with packages:
> > libvirt-0.10.2-24.el6.x86_64
> > qemu-kvm-0.12.1.2-2.400.el6.x86_64
> > kernel:2.6.32-416.el6.x86_64
> > 
> > The test last for over 48 hours, and don't hit this issue.
> 
> The test last for over 72 hours, and have not hit this issue.

The test last for over 100 hours, and have not hit this issue. So, I stop the test. The deadlock issue is no longer existed on the latest libvirt, qemu-kvm.
Note You need to log in before you can comment on or make changes to this bug.