Bug 499698

Can you run 'strace -f -p $PID-OF-LIBVIRTD -s 1000 -ff -o'   and then try and reproduce the destroy problem. 

Also, can you turn on full debug logging of libvirtd & capture the results http://libvirt.org/logging.html

Nasty. I've asked wwoods for a libvirtd log ala:

https://fedoraproject.org/wiki/Reporting_virtualization_bugs#libvirt

Created attachment 342908 [details]
very verbose log from libvirtd

This is the full log from libvirtd with log_level set to 1. It follows these basic steps:

service libvirtd restart 
virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
 13 F10                  running
  - F10_2                shut off
  - F10_RAID             shut off
  - F9                   shut off
  - Fedora_10_clone      shut off
  - Rawhide              shut off

virsh start F10_2
virsh start Fedora_10_clone
virsh list --all
virsh destroy F10_2
virsh list --all
virsh start F10_2
virsh destroy F10
virsh list --all
virsh start F10
virsh list --all
virsh list --all
virsh start F10_2
virsh list --all
# problem is triggered here - F10 dies as well
virsh destroy F10_2
virsh list --all
 Id Name                 State
----------------------------------
  1 Ubuntu_Jaunty        running
 15 Fedora_10_clone      running
  - F10                  shut off
  - F10_2                shut off
  - F10_RAID             shut off
  - F9                   shut off
  - Rawhide              shut off

service libvirtd stop

Hope you can make some sense of it - it's 4MB uncompressed.

Interesting, you only issued a destroy for F10_2, but yet:

14:12:43.927: debug : virDomainDestroy:1750 : domain=0x7fb270001270
14:12:43.927: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10_2'

14:12:43.927: debug : virEventRemoveHandleImpl:165 : Remove handle 20
14:12:43.927: debug : virEventRemoveHandleImpl:172 : mark delete 11 24
14:12:43.983: debug : virEventRunOnce:544 : Poll got 1 event
14:12:43.983: debug : virEventDispatchHandles:416 : Skip deleted 24
14:12:43.983: debug : virEventDispatchHandles:425 : Dispatch 24 32 0x13f7c00
14:12:44.036: info : Setting SELinux context on '/var/lib/libvirt/images/F10_2.img' to 'system_u:object_r:virt_image_t:s0'
14:12:44.036: debug : virEventUpdateTimeoutImpl:233 : Updating timer 0 timeout with 0 ms freq
14:12:44.036: debug : qemudShutdownVMDaemon:1518 : Shutting down VM 'F10'

This is an event loop dispatcher bug. The destroy call is killing the QEMU process, so we then get a HANGUP event on the FD associated with the guest monitor. The callback for this is already marked as deleted in the event loop though, so it gets skiped, and then we mistakenly dispatch the next callback in the loop, causing us to think another VM has died, and trigger cleanup of that guest

*** Bug 499788 has been marked as a duplicate of this bug. ***

Created attachment 343109 [details]
Fix event loop handling of deletes & test functionality

THis patch fixes the event loop handling of deletes and adds a test case which validates the various important scenarios actually work

*** Bug 500089 has been marked as a duplicate of this bug. ***

libvirt-0.6.2-9.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-9.fc11

I've pushed libvirt-0.6.2-9.fc11 to updates-testing with Dan's fix. Please test and update the update's karma using the link above

* Thu May 21 2009 Mark McLoughlin <markmc> - 0.6.2-9.fc11
- Fix qemu argv detection with latest qemu (bug #501923)
- Fix XML attribute escaping (bug #499791)
- Fix serious event handling issues causing guests to be destroyed (bug #499698)

libvirt-0.6.2-9.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5311

libvirt-0.6.2-10.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-10.fc11

libvirt-0.6.2-10.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5441

libvirt-0.6.2-11.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-5515

libvirt-0.6.2-11.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

libvirt-0.6.2-12.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-12.fc11