+++ This bug was initially created as a clone of Bug #499698 +++ Sometimes, using 'virsh destroy' or the 'Force off' button in virt-manager will cause multiple running VMs to be destroyed. Here's an example: [wwoods@metroid ~]$ sudo virsh list --all Id Name State ---------------------------------- 1 Ubuntu_Jaunty running 4 Fedora_10_clone running 5 F10_2 running 6 F10 running [wwoods@metroid ~]$ sudo virsh destroy F10_2 Domain F10_2 destroyed [wwoods@metroid ~]$ sudo virsh list --all Id Name State ---------------------------------- 1 Ubuntu_Jaunty running 4 Fedora_10_clone running - F10 shut off - F10_2 shut off Note that 'F10' is now also shut off, even though I didn't destroy it. This doesn't seem to happen every time, and it doesn't seem to be related to the names of the hosts being similar: [wwoods@metroid ~]$ sudo virsh list --all Id Name State ---------------------------------- 1 Ubuntu_Jaunty running 4 Fedora_10_clone running 8 F10_2 running - F10 shut off [wwoods@metroid ~]$ sudo virsh start F10 Domain F10 started [wwoods@metroid ~]$ sudo virsh destroy Fedora_10_clone Domain Fedora_10_clone destroyed [wwoods@metroid ~]$ sudo virsh list --all Id Name State ---------------------------------- 1 Ubuntu_Jaunty running 10 F10 running - F10_2 shut off - Fedora_10_clone shut off There are no relevant messages in syslog, other than the expected ones (e.g. "kernel: virbr0: port 3(vnet2) entering disabled state" as the host comes down). --- Additional comment from berrange on 2009-05-07 14:01:06 EDT --- Can you run 'strace -f -p $PID-OF-LIBVIRTD -s 1000 -ff -o' and then try and reproduce the destroy problem. Also, can you turn on full debug logging of libvirtd & capture the results http://libvirt.org/logging.html --- Additional comment from markmc on 2009-05-07 14:04:14 EDT --- Nasty. I've asked wwoods for a libvirtd log ala: https://fedoraproject.org/wiki/Reporting_virtualization_bugs#libvirt --- Additional comment from wwoods on 2009-05-07 14:29:41 EDT --- Created an attachment (id=342908) very verbose log from libvirtd This is the full log from libvirtd with log_level set to 1. It follows these basic steps: service libvirtd restart virsh list --all Id Name State ---------------------------------- 1 Ubuntu_Jaunty running 13 F10 running - F10_2 shut off - F10_RAID shut off - F9 shut off - Fedora_10_clone shut off - Rawhide shut off virsh start F10_2 virsh start Fedora_10_clone virsh list --all virsh destroy F10_2 virsh list --all virsh start F10_2 virsh destroy F10 virsh list --all virsh start F10 virsh list --all virsh list --all virsh start F10_2 virsh list --all # problem is triggered here - F10 dies as well virsh destroy F10_2 virsh list --all Id Name State ---------------------------------- 1 Ubuntu_Jaunty running 15 Fedora_10_clone running - F10 shut off - F10_2 shut off - F10_RAID shut off - F9 shut off - Rawhide shut off service libvirtd stop Hope you can make some sense of it - it's 4MB uncompressed. --- Additional comment from markmc on 2009-05-07 14:50:37 EDT --- Interesting, you only issued a destroy for F10_2, but yet: 14:12:43.927: debug : virDomainDestroy:1750 : domain=0x7fb270001270 14:12:43.927: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10_2' 14:12:43.927: debug : virEventRemoveHandleImpl:165 : Remove handle 20 14:12:43.927: debug : virEventRemoveHandleImpl:172 : mark delete 11 24 14:12:43.983: debug : virEventRunOnce:544 : Poll got 1 event 14:12:43.983: debug : virEventDispatchHandles:416 : Skip deleted 24 14:12:43.983: debug : virEventDispatchHandles:425 : Dispatch 24 32 0x13f7c00 14:12:44.036: info : Setting SELinux context on '/var/lib/libvirt/images/F10_2.img' to 'system_u:object_r:virt_image_t:s0' 14:12:44.036: debug : virEventUpdateTimeoutImpl:233 : Updating timer 0 timeout with 0 ms freq 14:12:44.036: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10' --- Additional comment from berrange on 2009-05-07 16:11:32 EDT --- This is an event loop dispatcher bug. The destroy call is killing the QEMU process, so we then get a HANGUP event on the FD associated with the guest monitor. The callback for this is already marked as deleted in the event loop though, so it gets skiped, and then we mistakenly dispatch the next callback in the loop, causing us to think another VM has died, and trigger cleanup of that guest --- Additional comment from berrange on 2009-05-08 05:43:39 EDT --- *** Bug 499788 has been marked as a duplicate of this bug. *** --- Additional comment from berrange on 2009-05-08 11:19:24 EDT --- Created an attachment (id=343109) Fix event loop handling of deletes & test functionality THis patch fixes the event loop handling of deletes and adds a test case which validates the various important scenarios actually work --- Additional comment from berrange on 2009-05-11 08:48:48 EDT --- *** Bug 500089 has been marked as a duplicate of this bug. ***
*** Bug 500089 has been marked as a duplicate of this bug. ***
libvirt-0.6.3-3.el5 has been built in dist-5E-qu-candidate with the fix, Daniel
Preliminary tests with 0.6.3-3.el5 from ovirt-latest suggest BZ500089 was fixed as well.
Start up all guest domains. [root@dhcp-66-70-85 ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 13 foo1 idle 14 foo2 idle 15 foo3 idle 16 test1 idle 17 test2 idle 18 test3 idle [root@dhcp-66-70-85 ~]# virsh destroy foo1 Domain foo1 destroyed [root@dhcp-66-70-85 ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 14 foo2 idle 15 foo3 idle 16 test1 idle 17 test2 idle 18 test3 idle - foo1 shut off [root@dhcp-66-70-85 ~]# virsh destroy foo2 Domain foo2 destroyed [root@dhcp-66-70-85 ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 15 foo3 idle 16 test1 idle 17 test2 idle 18 test3 idle - foo1 shut off - foo2 shut off [root@dhcp-66-70-85 ~]# virsh destroy test2 Domain test2 destroyed [root@dhcp-66-70-85 ~]# virsh destroy test3 Domain test3 destroyed [root@dhcp-66-70-85 ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 15 foo3 idle 16 test1 idle - foo1 shut off - foo2 shut off - test2 shut off - test3 shut off [root@dhcp-66-70-85 ~]# virsh start foo1 Domain foo1 started [root@dhcp-66-70-85 ~]# virsh destroy foo3 Domain foo3 destroyed [root@dhcp-66-70-85 ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 16 test1 idle 19 foo1 idle - foo2 shut off - foo3 shut off - test2 shut off - test3 shut off This bug has been verified in libvirt 0.6.3-3 on rhel-5.4. Fixed.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2009-1269.html