DescriptionKashyap Chamarthy
2013-05-10 08:17:04 UTC
Description of problem:
-----------------------
Force shutting down a guest with 'virsh destroy' fails, leaves a defunct qemu process. This defunt process also cannot be killed, leaving the system in an invalid state:
Version:
--------
$ uname -r ; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
3.10.0-0.rc0.git23.1.fc20.x86_64
qemu-kvm-1.4.1-1.fc19.x86_64
libvirt-daemon-kvm-1.0.5-2.fc19.x86_64
libguestfs-1.21.35-1.fc19.x86_64
How reproducible:
-----------------
Twice
Steps to Reproduce:
-------------------
1. Create a minimal Fedora guest ( w/ versions mentioned previously)
A simple script -- https://github.com/kashyapc/virt-scripts/blob/master/qcow2-create-generic.bash
2. Try to force shutdown the guest
# virsh destroy regular-guest
Test:
------
-> List the guests & try to start it:
=====
$ virsh list --all
Id Name State
----------------------------------------------------
3 regular-guest in shutdown
=====
$ virsh start regular-guest
error: Domain is already active
=====
-> Attempt to force destroy it fails due to 'unkown' reason:
=====
$ virsh destroy regular-guest
error: Failed to destroy domain regular-guest
error: An error occurred, but the cause is unknown
=====
-> From /var/log/libvirt/qemu/regular-guest.log
=====
[...]
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5 (label charserial0)
qemu: terminating on signal 15 from pid 1147
=====
-> Still in "in shutdown" state:
=====
$ virsh list --all
Id Name State
----------------------------------------------------
3 regular-guest in shutdown
=====
$ virsh start regular-guest
error: Domain is already active
=====
-> Attempt to force destroy it fails due to 'unkown' reason:
=====
$ virsh destroy regular-guest
error: Failed to destroy domain regular-guest
error: An error occurred, but the cause is unknown
=====
-> From /var/log/libvirt/qemu/regular-guest.log
=====
[...]
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5 (label charserial0)
qemu: terminating on signal 15 from pid 1147
=====
-> Still in "in shutdown" state:
=====
$ virsh list --all
Id Name State
----------------------------------------------------
3 regular-guest in shutdown
=====
-> grep for a qemu process
=====
$ ps -ef | grep -i qemu
qemu 4962 1 18 May09 ? 02:11:27 [qemu-system-x86] <defunct>
root 11414 1432 0 03:25 pts/1 00:00:00 grep --color=auto -i qemu
=====
-> Try to kill it & grep for the process again:
=====
$ kill -9 4962
=====
ps -ef | grep -i qemu
qemu 4962 1 18 May09 ? 02:11:27 [qemu-system-x86] <defunct>
root 11416 1432 0 03:25 pts/1 00:00:00 grep --color=auto -i qemu
=====
Actual results:
---------------
The guest is not powered off force fully, and leaves a defunct qemu process.
===
$ virsh destroy regular-guest
error: Failed to destroy domain regular-guest
error: An error occurred, but the cause is unknown
===
Expected results:
-----------------
(a) The guest should be force destroyed successfully, when a 'destroy' command is issued.
(b) A meaningful error should be thrown instead of 'unknown' error.
Additional info:
----------------
(1) To give some wider context, I was doing nested virtualization testing on Intel, and stumbled on this bug
- https://github.com/kashyapc/nested-virt-notes-intel-f18
(2) -> From /var/log/libvirt/qemu/regular-guest.log
=====
[...]
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5 (label charserial0)
qemu: terminating on signal 15 from pid 1147
=====
Comment 1Kashyap Chamarthy
2013-05-10 08:42:03 UTC
Created attachment 745993[details]
Libvirt debug logs when "virsh destroy" operation was performed
- Enable Libvirt debugging in /etc/libvirt/libvirtd.conf
log_level = 1
log_outputs = "1:file:/var/tmp/libvirtd.log"
- Restart libvirtd service
systemctl restart libvirtd.service
- Capture the logs
tail -f /var/tmp/libvirtd.log | tee guest-destroy-log-info.txt