Bug 961665 - [virsh] Attempt to force destroy a guest fails due to 'unknown' reason, leaves a defunct qemu process
Summary: [virsh] Attempt to force destroy a guest fails due to 'unknown' reason, leave...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-05-10 08:17 UTC by Kashyap Chamarthy
Modified: 2014-04-23 01:40 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-11 21:08:53 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Libvirt debug logs when "virsh destroy" operation was performed (103.74 KB, text/plain)
2013-05-10 08:42 UTC, Kashyap Chamarthy
no flags Details

Description Kashyap Chamarthy 2013-05-10 08:17:04 UTC
Description of problem:
-----------------------
Force shutting down a guest with 'virsh destroy' fails, leaves a defunct qemu process. This defunt process also cannot be killed, leaving the system in an invalid state:

Version:
--------

$ uname -r ; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
3.10.0-0.rc0.git23.1.fc20.x86_64
qemu-kvm-1.4.1-1.fc19.x86_64
libvirt-daemon-kvm-1.0.5-2.fc19.x86_64
libguestfs-1.21.35-1.fc19.x86_64


How reproducible:
-----------------
Twice


Steps to Reproduce:
-------------------

1. Create a minimal Fedora guest ( w/ versions mentioned previously)
   A simple script -- https://github.com/kashyapc/virt-scripts/blob/master/qcow2-create-generic.bash
2. Try to force shutdown the guest
   # virsh destroy regular-guest


Test:
------

-> List the guests & try to start it:
=====
$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     regular-guest                  in shutdown
=====
$ virsh start regular-guest
error: Domain is already active
=====

-> Attempt to force destroy it fails due to 'unkown' reason:
=====
$ virsh destroy regular-guest
error: Failed to destroy domain regular-guest
error: An error occurred, but the cause is unknown
=====

-> From /var/log/libvirt/qemu/regular-guest.log
=====
[...]
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5 (label charserial0)
qemu: terminating on signal 15 from pid 1147
=====

-> Still in "in shutdown" state:
=====
$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     regular-guest                  in shutdown
=====
$ virsh start regular-guest
error: Domain is already active
=====

-> Attempt to force destroy it fails due to 'unkown' reason:
=====
$ virsh destroy regular-guest
error: Failed to destroy domain regular-guest
error: An error occurred, but the cause is unknown
=====

-> From /var/log/libvirt/qemu/regular-guest.log
=====
[...]
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5 (label charserial0)
qemu: terminating on signal 15 from pid 1147
=====

-> Still in "in shutdown" state:
=====
$ virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     regular-guest                  in shutdown
=====

-> grep for a qemu process
=====
$ ps -ef | grep -i qemu
qemu      4962     1 18 May09 ?        02:11:27 [qemu-system-x86] <defunct>
root     11414  1432  0 03:25 pts/1    00:00:00 grep --color=auto -i qemu

=====

-> Try to kill it & grep for the process again:
=====
$ kill -9 4962
=====
ps -ef | grep -i qemu
qemu      4962     1 18 May09 ?        02:11:27 [qemu-system-x86] <defunct>
root     11416  1432  0 03:25 pts/1    00:00:00 grep --color=auto -i qemu

=====

Actual results:
---------------

The guest is not powered off force fully, and leaves a defunct qemu process.

===
$ virsh destroy regular-guest
error: Failed to destroy domain regular-guest
error: An error occurred, but the cause is unknown
===


Expected results:
-----------------

(a) The guest should be force destroyed successfully, when a 'destroy' command is issued.
(b) A meaningful error should be thrown instead of 'unknown' error.


Additional info:
----------------

(1) To give some wider context, I was doing nested virtualization testing on Intel, and stumbled on this bug
	-  https://github.com/kashyapc/nested-virt-notes-intel-f18


(2) -> From /var/log/libvirt/qemu/regular-guest.log
=====
[...]
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/5 (label charserial0)
qemu: terminating on signal 15 from pid 1147
=====

Comment 1 Kashyap Chamarthy 2013-05-10 08:42:03 UTC
Created attachment 745993 [details]
Libvirt debug logs when "virsh destroy" operation was performed

- Enable Libvirt debugging in /etc/libvirt/libvirtd.conf

	log_level = 1
	log_outputs = "1:file:/var/tmp/libvirtd.log"

- Restart libvirtd service

	systemctl restart libvirtd.service

- Capture the logs

        tail -f /var/tmp/libvirtd.log  | tee guest-destroy-log-info.txt

Comment 2 Cole Robinson 2013-06-11 22:30:42 UTC
I haven't seen this for a while, Kashyap are you still seeing this with latest F19?

Comment 3 Cole Robinson 2013-07-11 21:08:53 UTC
Closing as WORKSFORME


Note You need to log in before you can comment on or make changes to this bug.