Bug 1022627 - stale qemu processes left running in nova due to incorrect libguestfs cleanup
stale qemu processes left running in nova due to incorrect libguestfs cleanup
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
4.0
x86_64 Linux
low Severity medium
: z4
: 4.0
Assigned To: Solly Ross
Dafna Ron
storage
: Rebase, Reopened, ZStream
: 1022561 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-23 12:02 EDT by Dafna Ron
Modified: 2014-10-30 10:34 EDT (History)
13 users (show)

See Also:
Fixed In Version: openstack-nova-2013.2.3-1.el6ost
Doc Type: Bug Fix
Doc Text:
Cause: Previously, the Compute libguestfs did not perform tear down correctly in certain cases, such as when attempting to perform SSH key injection when no SSH keys were specified. Consequence: libguestfs qemu instances would remain running, even after the Compute instance with which they were associated was destroyed. If one attempted to kill them, they would become zombies that required a restart of nova-compute to kill. Fix: We now tear down libguestfs correctly. Result: libguestfs instances no longer are left running after the build stage of the main Compute instance.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-05-29 16:34:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (12.10 MB, application/x-gzip)
2013-10-23 12:02 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1243757 None None None Never
OpenStack gerrit 77914 None None None Never

  None (edit)
Description Dafna Ron 2013-10-23 12:02:48 EDT
Created attachment 815465 [details]
logs

Description of problem:

i am working witg glusterfs configured as cinder backend. 
when I run an instance I can see that there are 2 different qemu services launched 
when I delete the instance only one of the services is killed. 

Version-Release number of selected component (if applicable):

openstack-nova-compute-2013.2-0.25.rc1.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. configure gluster as cinder backend 
2. launch an instance from an image
3. run ps -elf |grep qemu
4. delete the instance 
5. run ps -elf |grep qemu 

Actual results:

there are 2 different qemu services run but only one is killed when we delete the instance 

Expected results:

not sure two services should be run
but if so, both should be killed when we delete the instance 

Additional info:


[root@cougar07 ~(keystone_admin)]# nova list 
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
| b2cda446-9ff2-405f-845a-422f5548738f | dafna | ACTIVE | None       | Running     | novanetwork=192.168.32.2 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------+
[root@cougar07 ~(keystone_admin)]# 
[root@cougar07 ~(keystone_admin)]# 
[root@cougar07 ~(keystone_admin)]# 
[root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu
2 S nova     28760 28700 10  80   0 - 212831 poll_s 18:30 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.28700 -initrd /var/tmp/.guestfs-162/initrd.28700 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.28700,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsNphmhH/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm
6 S qemu     29002     1 65  80   0 - 624337 futex_ 18:30 ?       00:00:09 /usr/libexec/qemu-kvm -name instance-0000004b -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b2cda446-9ff2-405f-845a-422f5548738f -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2-0.25.rc1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=b2cda446-9ff2-405f-845a-422f5548738f -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000004b.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:b0:3f:49,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.135:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
0 S root     29022 13613  0  80   0 - 25813 pipe_w 18:30 pts/2    00:00:00 grep qemu
[root@cougar07 ~(keystone_admin)]# nova delete b2cda446-9ff2-405f-845a-422f5548738f
[root@cougar07 ~(keystone_admin)]# nova list 

[root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu
2 S nova     28760 28700  3  80   0 - 212831 poll_s 18:30 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.28700 -initrd /var/tmp/.guestfs-162/initrd.28700 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.28700,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsNphmhH/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm
0 S root     29158 13613  0  80   0 - 25814 pipe_w 18:31 pts/2    00:00:00 grep qemu
Comment 1 Solly Ross 2013-11-05 15:47:45 EST
See also https://bugzilla.redhat.com/show_bug.cgi?id=1022561.  If these two bugs are indeed related (which, at a cursory glance it would seem so) it looks like one of the qemu processes is waiting on a "poll" call in the kernel.
Comment 2 Solly Ross 2013-11-06 16:21:45 EST
@dron: what version of gluster are you on?
Comment 3 Dafna Ron 2013-11-07 23:25:40 EST
glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64
Comment 4 Solly Ross 2013-11-08 11:50:00 EST
@dron: according to some people I spoke to, 3.4.0 has some issues with this kind of thing.  Can you try a different version and see if you still have the issue?
Comment 10 Solly Ross 2013-12-19 15:15:04 EST
could not reproduce outside of @dron's setup
Comment 11 Dafna Ron 2014-01-14 07:17:19 EST
I am not sure why this is closed for insufficient info when you are welcome to use my setup for testing... 
this is still reproducible - reopening the bug. 


[root@cougar06 ~(keystone_admin)]# nova list --host cougar06.scl.lab.tlv.redhat.com
+--------------------------------------+-------------------------------------------+--------+------------+-------------+---------------------------+
| ID                                   | Name                                      | Status | Task State | Power State | Networks                  |
+--------------------------------------+-------------------------------------------+--------+------------+-------------+---------------------------+
| 3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 | rhel-3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 | ACTIVE | None       | Running     | novanetwork=192.168.32.11 |
| 727ab3c1-2501-4389-9dd0-ffac91d6e598 | rhel-727ab3c1-2501-4389-9dd0-ffac91d6e598 | ACTIVE | None       | Running     | novanetwork=192.168.32.10 |
| d3e9b5c9-08d2-4078-8909-829065ed3f5c | rhel-d3e9b5c9-08d2-4078-8909-829065ed3f5c | ACTIVE | None       | Running     | novanetwork=192.168.32.3  |
| e7548f4b-dd6f-45f5-acf7-90f0539156eb | rhel-e7548f4b-dd6f-45f5-acf7-90f0539156eb | ACTIVE | None       | Running     | novanetwork=192.168.32.2  |
| ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 | rhel-ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 | ACTIVE | None       | Running     | novanetwork=192.168.32.6  |
+--------------------------------------+-------------------------------------------+--------+------------+-------------+---------------------------+
[root@cougar06 ~(keystone_admin)]# ps -elf |grep qemu
2 S nova     12339 16028  0  80   0 - 212834 poll_s 14:11 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/e7548f4b-dd6f-45f5-acf7-90f0539156eb/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfseSsvUg/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb
2 S nova     12388 16028  0  80   0 - 212834 poll_s 14:11 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsPoXKwK/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb
6 S qemu     12641     1 11  80   0 - 1372140 poll_s 14:11 ?      00:00:28 /usr/libexec/qemu-kvm -name instance-00000001 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid e7548f4b-dd6f-45f5-acf7-90f0539156eb -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=e7548f4b-dd6f-45f5-acf7-90f0539156eb -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/e7548f4b-dd6f-45f5-acf7-90f0539156eb/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:e9:fc:a4,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/e7548f4b-dd6f-45f5-acf7-90f0539156eb/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
2 S nova     12678 16028  0  80   0 - 212834 poll_s 14:11 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/d3e9b5c9-08d2-4078-8909-829065ed3f5c/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfslQv6tn/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb
6 S qemu     12910     1 11  80   0 - 1371912 poll_s 14:11 ?      00:00:27 /usr/libexec/qemu-kvm -name instance-00000007 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000007.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:85:b1:0a,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:1 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
2 S nova     12951 16028  0  80   0 - 212834 poll_s 14:11 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/3eac27cb-5120-4ab4-a34f-5a98c4c62aa1/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsS3KqC9/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb
6 S qemu     13189     1 11  80   0 - 1371912 poll_s 14:11 ?      00:00:28 /usr/libexec/qemu-kvm -name instance-00000005 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d3e9b5c9-08d2-4078-8909-829065ed3f5c -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=d3e9b5c9-08d2-4078-8909-829065ed3f5c -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000005.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/d3e9b5c9-08d2-4078-8909-829065ed3f5c/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:a8:0d:db,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/d3e9b5c9-08d2-4078-8909-829065ed3f5c/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:2 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
2 S nova     13231 16028  0  80   0 - 212834 poll_s 14:11 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/727ab3c1-2501-4389-9dd0-ffac91d6e598/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsx5cXc5/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb
6 S qemu     13460     1 12  80   0 - 1372581 poll_s 14:11 ?      00:00:29 /usr/libexec/qemu-kvm -name instance-00000009 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000009.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/3eac27cb-5120-4ab4-a34f-5a98c4c62aa1/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:5f:45:7a,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/3eac27cb-5120-4ab4-a34f-5a98c4c62aa1/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:3 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
6 S qemu     13715     1 12  80   0 - 1371912 poll_s 14:12 ?      00:00:28 /usr/libexec/qemu-kvm -name instance-00000003 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 727ab3c1-2501-4389-9dd0-ffac91d6e598 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=727ab3c1-2501-4389-9dd0-ffac91d6e598 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/727ab3c1-2501-4389-9dd0-ffac91d6e598/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:42:ae:52,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/727ab3c1-2501-4389-9dd0-ffac91d6e598/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:4 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
0 S root     15121  6091  0  80   0 - 25814 pipe_w 14:15 pts/0    00:00:00 grep qemu
Comment 12 Dave Allan 2014-01-14 15:29:08 EST
(In reply to Dafna Ron from comment #11)
> I am not sure why this is closed for insufficient info when you are welcome
> to use my setup for testing... 
> this is still reproducible - reopening the bug. 

There are innumerable reasons why problems might exist on a system that are not the result of bugs in the product.  If you can reproduce this behavior on a newly installed system and provide steps to reproduce, please reopen it, but otherwise we have done all we can here.
Comment 13 Dafna Ron 2014-01-14 15:34:45 EST
(In reply to Dave Allan from comment #12)
> (In reply to Dafna Ron from comment #11)
> > I am not sure why this is closed for insufficient info when you are welcome
> > to use my setup for testing... 
> > this is still reproducible - reopening the bug. 
> 
> There are innumerable reasons why problems might exist on a system that are
> not the result of bugs in the product.  If you can reproduce this behavior
> on a newly installed system and provide steps to reproduce, please reopen
> it, but otherwise we have done all we can here.

I am reproducing this on every new system I install (and I ruin one pretty much on a weekly basis :)) 
My previous comment and info was from a system I installed today using the latest build. 
Steps to reproduce have not changed... I am not sure why you cannot reproduce yourselves but I am available to try and see what is the difference between my installation/HW/steps/setup from the ones you are using.
Comment 14 Solly Ross 2014-01-17 16:07:08 EST
I have diagnosed the issue.  It only occurs when you launch a VM with no keypairs registered, and has to do with the libguestfs VFS driver.  The issue is caused by
nova/virt/disk/api:179 and nova/virt/disk/vfs/guestfs.py:136 -- namely, that in case of an error in VFSGuestFS#setup(), VFSGuestFS#teardown() is not called.  Instead, `handle` is set to `None`, supposedly calling the implicit teardown methods on `handle`.  However, this is not sufficient, and leaves the libguestfs VM running.

Calling an explicit teardown on error in setup seems to fix the issue.

I'm submitting an upstream bug and fix, and will backport it.
Comment 15 Solly Ross 2014-02-06 16:35:06 EST
upstream review in progress
Comment 21 Solly Ross 2014-04-07 11:02:53 EDT
*** Bug 1022561 has been marked as a duplicate of this bug. ***
Comment 23 Dafna Ron 2014-04-17 13:17:31 EDT
verified on openstack-nova-compute-2013.2.3-5.el6ost.noarch
Comment 25 errata-xmlrpc 2014-05-29 16:34:57 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0578.html
Comment 26 Haitao Jiang 2014-10-28 20:10:41 EDT
(In reply to Solly Ross from comment #14)
> I have diagnosed the issue.  It only occurs when you launch a VM with no
> keypairs registered, and has to do with the libguestfs VFS driver.  The
> issue is caused by
> nova/virt/disk/api:179 and nova/virt/disk/vfs/guestfs.py:136 -- namely, that
> in case of an error in VFSGuestFS#setup(), VFSGuestFS#teardown() is not
> called.  Instead, `handle` is set to `None`, supposedly calling the implicit
> teardown methods on `handle`.  However, this is not sufficient, and leaves
> the libguestfs VM running.
> 
> Calling an explicit teardown on error in setup seems to fix the issue.
> 
> I'm submitting an upstream bug and fix, and will backport it.

Just found this. I am having the same issue openstack-nova-compute.2013.2.2.1.el6. To re-produce, just boot a VM and then delete it. There will be one qemu process left behind. Tried to use --key-name <registered keypair name> option but still the same thing.
Comment 27 Solly Ross 2014-10-30 10:34:44 EDT
2013.2.2-1 does not contain the backport of the fix.  Please try 2013.2.4-1 instead, and see if you can still reproduce the issue.

Note You need to log in before you can comment on or make changes to this bug.