Bug 995795 - Deadlock on libvirt when playing with hotplug and add/remove vm.
Deadlock on libvirt when playing with hotplug and add/remove vm.
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.5
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Martin Kletzander
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-11 03:49 EDT by chhu
Modified: 2016-04-26 10:10 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-26 09:44:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description chhu 2013-08-11 03:49:20 EDT
Description of problem:
Deadlock on libvirt when playing with hotplug and add/remove vm.

Version-Release number of selected component (if applicable):
libvirt-0.10.2-21.el6.x86_64
qemu-kvm-0.12.1.2-2.385.el6.x86_64
2.6.32-396.el6.x86_64

How reproducible:
80%


1. Prepare and start a guest.

2. Open the first terminal and run the following commands(add your domain name and image name):
# while true; do virsh attach-disk <domain> /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb; sleep 2; done

3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images' name from r1 to r10, and with guests' name from r1 to r10. Open the second terminal and run the following commands:

# while true; do for i in {1..10}; do virsh create /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy r$i; done; done

4. Wait for more than 10 hours

5. Deadlock on libvirt
# virsh list --all
error: Failed to reconnect to the hypervisor
error: no valid connection
error: Cannot recv data: Connection reset by peer

6. Check the libvirtd service
# service libvirtd status
libvirtd (pid  6757) is running...


Actual results:
Met deadlock, "Failed to reconnect to the hypervisor".

Expected results:
libvirt work well
Comment 2 Martin Kletzander 2013-08-13 08:16:58 EDT
What is the state libvirtd ends up in?  Please provide a backtrace of the process obtained by issuing a 't a a bt full' command in attached gdb.  Also provide a debug log from the same time when the backtrace was extracted.  Thank you.
Comment 6 chhu 2013-09-12 04:04:49 EDT
(In reply to chhu from comment #0)
> Description of problem:
> Deadlock on libvirt when playing with hotplug and add/remove vm.
> 
> Version-Release number of selected component (if applicable):
> libvirt-0.10.2-21.el6.x86_64
> qemu-kvm-0.12.1.2-2.385.el6.x86_64
> 2.6.32-396.el6.x86_64
> 
> How reproducible:
> 80%
> 
> 
> 1. Prepare and start a guest.
> 
> 2. Open the first terminal and run the following commands(add your domain
> name and image name):
> # while true; do virsh attach-disk <domain>
> /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb;
> sleep 2; done
> 
> 3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images'
> name from r1 to r10, and with guests' name from r1 to r10. Open the second
> terminal and run the following commands:
> 
> # while true; do for i in {1..10}; do virsh create
> /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy
> r$i; done; done
> 
> 4. Wait for more than 10 hours
> 
> 5. Deadlock on libvirt
> # virsh list --all
> error: Failed to reconnect to the hypervisor
> error: no valid connection
> error: Cannot recv data: Connection reset by peer
> 
> 6. Check the libvirtd service
> # service libvirtd status
> libvirtd (pid  6757) is running...
> 
> 
> Actual results:
> Met deadlock, "Failed to reconnect to the hypervisor".
> 
> Expected results:
> libvirt work well


Rerun the test with packages:
libvirt-0.10.2-24.el6.x86_64
qemu-kvm-0.12.1.2-2.400.el6.x86_64
kernel:2.6.32-416.el6.x86_64

The test last for over 48 hours, and don't hit this issue.
Comment 7 chhu 2013-09-13 04:16:13 EDT
(In reply to chhu from comment #6)
> (In reply to chhu from comment #0)
> > Description of problem:
> > Deadlock on libvirt when playing with hotplug and add/remove vm.
> > 
> > Version-Release number of selected component (if applicable):
> > libvirt-0.10.2-21.el6.x86_64
> > qemu-kvm-0.12.1.2-2.385.el6.x86_64
> > 2.6.32-396.el6.x86_64
> > 
> > How reproducible:
> > 80%
> > 
> > 
> > 1. Prepare and start a guest.
> > 
> > 2. Open the first terminal and run the following commands(add your domain
> > name and image name):
> > # while true; do virsh attach-disk <domain>
> > /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb;
> > sleep 2; done
> > 
> > 3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images'
> > name from r1 to r10, and with guests' name from r1 to r10. Open the second
> > terminal and run the following commands:
> > 
> > # while true; do for i in {1..10}; do virsh create
> > /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy
> > r$i; done; done
> > 
> > 4. Wait for more than 10 hours
> > 
> > 5. Deadlock on libvirt
> > # virsh list --all
> > error: Failed to reconnect to the hypervisor
> > error: no valid connection
> > error: Cannot recv data: Connection reset by peer
> > 
> > 6. Check the libvirtd service
> > # service libvirtd status
> > libvirtd (pid  6757) is running...
> > 
> > 
> > Actual results:
> > Met deadlock, "Failed to reconnect to the hypervisor".
> > 
> > Expected results:
> > libvirt work well
> 
> 
> Rerun the test with packages:
> libvirt-0.10.2-24.el6.x86_64
> qemu-kvm-0.12.1.2-2.400.el6.x86_64
> kernel:2.6.32-416.el6.x86_64
> 
> The test last for over 48 hours, and don't hit this issue.

The test last for over 72 hours, and have not hit this issue.
Comment 8 chhu 2013-09-15 23:11:38 EDT
(In reply to chhu from comment #7)
> (In reply to chhu from comment #6)
> > (In reply to chhu from comment #0)
> > > Description of problem:
> > > Deadlock on libvirt when playing with hotplug and add/remove vm.
> > > 
> > > Version-Release number of selected component (if applicable):
> > > libvirt-0.10.2-21.el6.x86_64
> > > qemu-kvm-0.12.1.2-2.385.el6.x86_64
> > > 2.6.32-396.el6.x86_64
> > > 
> > > How reproducible:
> > > 80%
> > > 
> > > 
> > > 1. Prepare and start a guest.
> > > 
> > > 2. Open the first terminal and run the following commands(add your domain
> > > name and image name):
> > > # while true; do virsh attach-disk <domain>
> > > /var/lib/libvirt/images/<image>.img vdb;  virsh detach-disk <domain> vdb;
> > > sleep 2; done
> > > 
> > > 3. Prepare another 10 guests with xml files from r1.xml to r10.xml, images'
> > > name from r1 to r10, and with guests' name from r1 to r10. Open the second
> > > terminal and run the following commands:
> > > 
> > > # while true; do for i in {1..10}; do virsh create
> > > /var/lib/libvirt/images/r$i.xml; done; for i in {1..10}; do virsh destroy
> > > r$i; done; done
> > > 
> > > 4. Wait for more than 10 hours
> > > 
> > > 5. Deadlock on libvirt
> > > # virsh list --all
> > > error: Failed to reconnect to the hypervisor
> > > error: no valid connection
> > > error: Cannot recv data: Connection reset by peer
> > > 
> > > 6. Check the libvirtd service
> > > # service libvirtd status
> > > libvirtd (pid  6757) is running...
> > > 
> > > 
> > > Actual results:
> > > Met deadlock, "Failed to reconnect to the hypervisor".
> > > 
> > > Expected results:
> > > libvirt work well
> > 
> > 
> > Rerun the test with packages:
> > libvirt-0.10.2-24.el6.x86_64
> > qemu-kvm-0.12.1.2-2.400.el6.x86_64
> > kernel:2.6.32-416.el6.x86_64
> > 
> > The test last for over 48 hours, and don't hit this issue.
> 
> The test last for over 72 hours, and have not hit this issue.

The test last for over 100 hours, and have not hit this issue. So, I stop the test. The deadlock issue is no longer existed on the latest libvirt, qemu-kvm.

Note You need to log in before you can comment on or make changes to this bug.