Bug 499698 - 'virsh destroy' destroys multiple VMs
'virsh destroy' destroys multiple VMs
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
rawhide
All Linux
high Severity high
: ---
: ---
Assigned To: Daniel Berrange
Fedora Extras Quality Assurance
:
: 499788 (view as bug list)
Depends On:
Blocks: F11VirtTarget 500158
  Show dependency treegraph
 
Reported: 2009-05-07 13:48 EDT by Will Woods
Modified: 2009-06-05 09:24 EDT (History)
10 users (show)

See Also:
Fixed In Version: 0.6.2-11.fc11
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 500158 (view as bug list)
Environment:
Last Closed: 2009-06-04 17:21:45 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
very verbose log from libvirtd (84.85 KB, application/x-bzip)
2009-05-07 14:29 EDT, Will Woods
no flags Details
Fix event loop handling of deletes & test functionality (27.44 KB, patch)
2009-05-08 11:19 EDT, Daniel Berrange
no flags Details | Diff

  None (edit)
Description Will Woods 2009-05-07 13:48:57 EDT
Sometimes, using 'virsh destroy' or the 'Force off' button in virt-manager will cause multiple running VMs to be destroyed. Here's an example:

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
  4 Fedora_10_clone      running
  5 F10_2                running
  6 F10                  running

[wwoods@metroid ~]$ sudo virsh destroy F10_2
Domain F10_2 destroyed

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
  4 Fedora_10_clone      running
  - F10                  shut off
  - F10_2                shut off

Note that 'F10' is now also shut off, even though I didn't destroy it. This doesn't seem to happen every time, and it doesn't seem to be related to the names of the hosts being similar:

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
  4 Fedora_10_clone      running
  8 F10_2                running
  - F10                  shut off

[wwoods@metroid ~]$ sudo virsh start F10
Domain F10 started

[wwoods@metroid ~]$ sudo virsh destroy Fedora_10_clone
Domain Fedora_10_clone destroyed

[wwoods@metroid ~]$ sudo virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
 10 F10                  running
  - F10_2                shut off
  - Fedora_10_clone      shut off

There are no relevant messages in syslog, other than the expected ones (e.g. "kernel: virbr0: port 3(vnet2) entering disabled state" as the host comes down).
Comment 1 Daniel Berrange 2009-05-07 14:01:06 EDT
Can you run 'strace -f -p $PID-OF-LIBVIRTD -s 1000 -ff -o'   and then try and reproduce the destroy problem. 

Also, can you turn on full debug logging of libvirtd & capture the results http://libvirt.org/logging.html
Comment 2 Mark McLoughlin 2009-05-07 14:04:14 EDT
Nasty. I've asked wwoods for a libvirtd log ala:

https://fedoraproject.org/wiki/Reporting_virtualization_bugs#libvirt
Comment 3 Will Woods 2009-05-07 14:29:41 EDT
Created attachment 342908 [details]
very verbose log from libvirtd

This is the full log from libvirtd with log_level set to 1. It follows these basic steps:

service libvirtd restart 
virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
 13 F10                  running
  - F10_2                shut off
  - F10_RAID             shut off
  - F9                   shut off
  - Fedora_10_clone      shut off
  - Rawhide              shut off

virsh start F10_2
virsh start Fedora_10_clone
virsh list --all
virsh destroy F10_2
virsh list --all
virsh start F10_2
virsh destroy F10
virsh list --all
virsh start F10
virsh list --all
virsh list --all
virsh start F10_2
virsh list --all
# problem is triggered here - F10 dies as well
virsh destroy F10_2
virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
 15 Fedora_10_clone      running
  - F10                  shut off
  - F10_2                shut off
  - F10_RAID             shut off
  - F9                   shut off
  - Rawhide              shut off

service libvirtd stop

Hope you can make some sense of it - it's 4MB uncompressed.
Comment 4 Mark McLoughlin 2009-05-07 14:50:37 EDT
Interesting, you only issued a destroy for F10_2, but yet:

14:12:43.927: debug : virDomainDestroy:1750 : domain=0x7fb270001270
14:12:43.927: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10_2'

14:12:43.927: debug : virEventRemoveHandleImpl:165 : Remove handle 20
14:12:43.927: debug : virEventRemoveHandleImpl:172 : mark delete 11 24
14:12:43.983: debug : virEventRunOnce:544 : Poll got 1 event
14:12:43.983: debug : virEventDispatchHandles:416 : Skip deleted 24
14:12:43.983: debug : virEventDispatchHandles:425 : Dispatch 24 32 0x13f7c00
14:12:44.036: info : Setting SELinux context on '/var/lib/libvirt/images/F10_2.img' to 'system_u:object_r:virt_image_t:s0'
14:12:44.036: debug : virEventUpdateTimeoutImpl:233 : Updating timer 0 timeout with 0 ms freq
14:12:44.036: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10'
Comment 5 Daniel Berrange 2009-05-07 16:11:32 EDT
This is an event loop dispatcher bug. The destroy call is killing the QEMU process, so we then get a HANGUP event on the FD associated with the guest monitor. The callback for this is already marked as deleted in the event loop though, so it gets skiped, and then we mistakenly dispatch the next callback in the loop, causing us to think another VM has died, and trigger cleanup of that guest
Comment 6 Daniel Berrange 2009-05-08 05:43:39 EDT
*** Bug 499788 has been marked as a duplicate of this bug. ***
Comment 7 Daniel Berrange 2009-05-08 11:19:24 EDT
Created attachment 343109 [details]
Fix event loop handling of deletes & test functionality

THis patch fixes the event loop handling of deletes and adds a test case which validates the various important scenarios actually work
Comment 8 Daniel Berrange 2009-05-11 08:48:48 EDT
*** Bug 500089 has been marked as a duplicate of this bug. ***
Comment 9 Fedora Update System 2009-05-21 09:25:24 EDT
libvirt-0.6.2-9.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-9.fc11
Comment 10 Mark McLoughlin 2009-05-21 09:30:58 EDT
I've pushed libvirt-0.6.2-9.fc11 to updates-testing with Dan's fix. Please test and update the update's karma using the link above

* Thu May 21 2009 Mark McLoughlin <markmc@redhat.com> - 0.6.2-9.fc11
- Fix qemu argv detection with latest qemu (bug #501923)
- Fix XML attribute escaping (bug #499791)
- Fix serious event handling issues causing guests to be destroyed (bug #499698)
Comment 11 Fedora Update System 2009-05-21 19:28:11 EDT
libvirt-0.6.2-9.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5311
Comment 12 Fedora Update System 2009-05-22 05:53:28 EDT
libvirt-0.6.2-10.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-10.fc11
Comment 13 Fedora Update System 2009-05-25 17:20:14 EDT
libvirt-0.6.2-10.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5441
Comment 14 Fedora Update System 2009-05-26 03:57:53 EDT
libvirt-0.6.2-11.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5515
Comment 15 Fedora Update System 2009-06-04 17:21:19 EDT
libvirt-0.6.2-11.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 16 Fedora Update System 2009-06-05 09:24:10 EDT
libvirt-0.6.2-12.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-12.fc11

Note You need to log in before you can comment on or make changes to this bug.