Description of problem: ----------------------- Force shutting down a guest with 'virsh destroy' fails, leaves a defunct qemu process. This defunt process also cannot be killed, leaving the system in an invalid state: Version: -------- $ uname -r ; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs 3.10.0-0.rc0.git23.1.fc20.x86_64 qemu-kvm-1.4.1-1.fc19.x86_64 libvirt-daemon-kvm-1.0.5-2.fc19.x86_64 libguestfs-1.21.35-1.fc19.x86_64 How reproducible: ----------------- Twice Steps to Reproduce: ------------------- 1. Create a minimal Fedora guest ( w/ versions mentioned previously) A simple script -- https://github.com/kashyapc/virt-scripts/blob/master/qcow2-create-generic.bash 2. Try to force shutdown the guest # virsh destroy regular-guest Test: ------ -> List the guests & try to start it: ===== $ virsh list --all Id Name State ---------------------------------------------------- 3 regular-guest in shutdown ===== $ virsh start regular-guest error: Domain is already active ===== -> Attempt to force destroy it fails due to 'unkown' reason: ===== $ virsh destroy regular-guest error: Failed to destroy domain regular-guest error: An error occurred, but the cause is unknown ===== -> From /var/log/libvirt/qemu/regular-guest.log ===== [...] LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/5 (label charserial0) qemu: terminating on signal 15 from pid 1147 ===== -> Still in "in shutdown" state: ===== $ virsh list --all Id Name State ---------------------------------------------------- 3 regular-guest in shutdown ===== $ virsh start regular-guest error: Domain is already active ===== -> Attempt to force destroy it fails due to 'unkown' reason: ===== $ virsh destroy regular-guest error: Failed to destroy domain regular-guest error: An error occurred, but the cause is unknown ===== -> From /var/log/libvirt/qemu/regular-guest.log ===== [...] LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/5 (label charserial0) qemu: terminating on signal 15 from pid 1147 ===== -> Still in "in shutdown" state: ===== $ virsh list --all Id Name State ---------------------------------------------------- 3 regular-guest in shutdown ===== -> grep for a qemu process ===== $ ps -ef | grep -i qemu qemu 4962 1 18 May09 ? 02:11:27 [qemu-system-x86] <defunct> root 11414 1432 0 03:25 pts/1 00:00:00 grep --color=auto -i qemu ===== -> Try to kill it & grep for the process again: ===== $ kill -9 4962 ===== ps -ef | grep -i qemu qemu 4962 1 18 May09 ? 02:11:27 [qemu-system-x86] <defunct> root 11416 1432 0 03:25 pts/1 00:00:00 grep --color=auto -i qemu ===== Actual results: --------------- The guest is not powered off force fully, and leaves a defunct qemu process. === $ virsh destroy regular-guest error: Failed to destroy domain regular-guest error: An error occurred, but the cause is unknown === Expected results: ----------------- (a) The guest should be force destroyed successfully, when a 'destroy' command is issued. (b) A meaningful error should be thrown instead of 'unknown' error. Additional info: ---------------- (1) To give some wider context, I was doing nested virtualization testing on Intel, and stumbled on this bug - https://github.com/kashyapc/nested-virt-notes-intel-f18 (2) -> From /var/log/libvirt/qemu/regular-guest.log ===== [...] LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 char device redirected to /dev/pts/5 (label charserial0) qemu: terminating on signal 15 from pid 1147 =====
Created attachment 745993 [details] Libvirt debug logs when "virsh destroy" operation was performed - Enable Libvirt debugging in /etc/libvirt/libvirtd.conf log_level = 1 log_outputs = "1:file:/var/tmp/libvirtd.log" - Restart libvirtd service systemctl restart libvirtd.service - Capture the logs tail -f /var/tmp/libvirtd.log | tee guest-destroy-log-info.txt
I haven't seen this for a while, Kashyap are you still seeing this with latest F19?
Closing as WORKSFORME