Bug 1022627
| Summary: | stale qemu processes left running in nova due to incorrect libguestfs cleanup | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Dafna Ron <dron> | ||||
| Component: | openstack-nova | Assignee: | Solly Ross <sross> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Dafna Ron <dron> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 4.0 | CC: | breeler, dallan, dron, gfidente, hateya, jianghaitao, ndipanov, nlevinki, sgordon, sross, xqueralt, yeylon | ||||
| Target Milestone: | z4 | Keywords: | Rebase, Reopened, ZStream | ||||
| Target Release: | 4.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | storage | ||||||
| Fixed In Version: | openstack-nova-2013.2.3-1.el6ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
Cause: Previously, the Compute libguestfs did not perform tear down correctly in certain cases, such as when attempting to perform SSH key injection when no SSH keys were specified.
Consequence: libguestfs qemu instances would remain running, even after the Compute instance with which they were associated was destroyed. If one attempted to kill them, they would become zombies that required a restart of nova-compute to kill.
Fix: We now tear down libguestfs correctly.
Result: libguestfs instances no longer are left running after the build stage of the main Compute instance.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-05-29 20:34:57 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
See also https://bugzilla.redhat.com/show_bug.cgi?id=1022561. If these two bugs are indeed related (which, at a cursory glance it would seem so) it looks like one of the qemu processes is waiting on a "poll" call in the kernel. @dron: what version of gluster are you on? glusterfs-fuse-3.4.0.33rhs-1.el6rhs.x86_64 @dron: according to some people I spoke to, 3.4.0 has some issues with this kind of thing. Can you try a different version and see if you still have the issue? could not reproduce outside of @dron's setup I am not sure why this is closed for insufficient info when you are welcome to use my setup for testing... this is still reproducible - reopening the bug. [root@cougar06 ~(keystone_admin)]# nova list --host cougar06.scl.lab.tlv.redhat.com +--------------------------------------+-------------------------------------------+--------+------------+-------------+---------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------------------------+--------+------------+-------------+---------------------------+ | 3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 | rhel-3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 | ACTIVE | None | Running | novanetwork=192.168.32.11 | | 727ab3c1-2501-4389-9dd0-ffac91d6e598 | rhel-727ab3c1-2501-4389-9dd0-ffac91d6e598 | ACTIVE | None | Running | novanetwork=192.168.32.10 | | d3e9b5c9-08d2-4078-8909-829065ed3f5c | rhel-d3e9b5c9-08d2-4078-8909-829065ed3f5c | ACTIVE | None | Running | novanetwork=192.168.32.3 | | e7548f4b-dd6f-45f5-acf7-90f0539156eb | rhel-e7548f4b-dd6f-45f5-acf7-90f0539156eb | ACTIVE | None | Running | novanetwork=192.168.32.2 | | ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 | rhel-ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 | ACTIVE | None | Running | novanetwork=192.168.32.6 | +--------------------------------------+-------------------------------------------+--------+------------+-------------+---------------------------+ [root@cougar06 ~(keystone_admin)]# ps -elf |grep qemu 2 S nova 12339 16028 0 80 0 - 212834 poll_s 14:11 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/e7548f4b-dd6f-45f5-acf7-90f0539156eb/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfseSsvUg/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb 2 S nova 12388 16028 0 80 0 - 212834 poll_s 14:11 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsPoXKwK/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb 6 S qemu 12641 1 11 80 0 - 1372140 poll_s 14:11 ? 00:00:28 /usr/libexec/qemu-kvm -name instance-00000001 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid e7548f4b-dd6f-45f5-acf7-90f0539156eb -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=e7548f4b-dd6f-45f5-acf7-90f0539156eb -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/e7548f4b-dd6f-45f5-acf7-90f0539156eb/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:e9:fc:a4,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/e7548f4b-dd6f-45f5-acf7-90f0539156eb/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 2 S nova 12678 16028 0 80 0 - 212834 poll_s 14:11 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/d3e9b5c9-08d2-4078-8909-829065ed3f5c/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfslQv6tn/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb 6 S qemu 12910 1 11 80 0 - 1371912 poll_s 14:11 ? 00:00:27 /usr/libexec/qemu-kvm -name instance-00000007 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000007.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:85:b1:0a,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/ffa461cc-3e0f-45d9-9a85-f0e2cd8ce818/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:1 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 2 S nova 12951 16028 0 80 0 - 212834 poll_s 14:11 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/3eac27cb-5120-4ab4-a34f-5a98c4c62aa1/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsS3KqC9/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb 6 S qemu 13189 1 11 80 0 - 1371912 poll_s 14:11 ? 00:00:28 /usr/libexec/qemu-kvm -name instance-00000005 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d3e9b5c9-08d2-4078-8909-829065ed3f5c -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=d3e9b5c9-08d2-4078-8909-829065ed3f5c -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000005.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/d3e9b5c9-08d2-4078-8909-829065ed3f5c/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:a8:0d:db,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/d3e9b5c9-08d2-4078-8909-829065ed3f5c/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:2 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 2 S nova 13231 16028 0 80 0 - 212834 poll_s 14:11 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.16028 -initrd /var/tmp/.guestfs-162/initrd.16028 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/727ab3c1-2501-4389-9dd0-ffac91d6e598/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.16028,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsx5cXc5/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=dumb 6 S qemu 13460 1 12 80 0 - 1372581 poll_s 14:11 ? 00:00:29 /usr/libexec/qemu-kvm -name instance-00000009 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=3eac27cb-5120-4ab4-a34f-5a98c4c62aa1 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000009.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/3eac27cb-5120-4ab4-a34f-5a98c4c62aa1/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:5f:45:7a,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/3eac27cb-5120-4ab4-a34f-5a98c4c62aa1/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:3 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 6 S qemu 13715 1 12 80 0 - 1371912 poll_s 14:12 ? 00:00:28 /usr/libexec/qemu-kvm -name instance-00000003 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 727ab3c1-2501-4389-9dd0-ffac91d6e598 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2.1-1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=727ab3c1-2501-4389-9dd0-ffac91d6e598 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000003.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/727ab3c1-2501-4389-9dd0-ffac91d6e598/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:42:ae:52,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/727ab3c1-2501-4389-9dd0-ffac91d6e598/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.133:4 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 0 S root 15121 6091 0 80 0 - 25814 pipe_w 14:15 pts/0 00:00:00 grep qemu (In reply to Dafna Ron from comment #11) > I am not sure why this is closed for insufficient info when you are welcome > to use my setup for testing... > this is still reproducible - reopening the bug. There are innumerable reasons why problems might exist on a system that are not the result of bugs in the product. If you can reproduce this behavior on a newly installed system and provide steps to reproduce, please reopen it, but otherwise we have done all we can here. (In reply to Dave Allan from comment #12) > (In reply to Dafna Ron from comment #11) > > I am not sure why this is closed for insufficient info when you are welcome > > to use my setup for testing... > > this is still reproducible - reopening the bug. > > There are innumerable reasons why problems might exist on a system that are > not the result of bugs in the product. If you can reproduce this behavior > on a newly installed system and provide steps to reproduce, please reopen > it, but otherwise we have done all we can here. I am reproducing this on every new system I install (and I ruin one pretty much on a weekly basis :)) My previous comment and info was from a system I installed today using the latest build. Steps to reproduce have not changed... I am not sure why you cannot reproduce yourselves but I am available to try and see what is the difference between my installation/HW/steps/setup from the ones you are using. I have diagnosed the issue. It only occurs when you launch a VM with no keypairs registered, and has to do with the libguestfs VFS driver. The issue is caused by nova/virt/disk/api:179 and nova/virt/disk/vfs/guestfs.py:136 -- namely, that in case of an error in VFSGuestFS#setup(), VFSGuestFS#teardown() is not called. Instead, `handle` is set to `None`, supposedly calling the implicit teardown methods on `handle`. However, this is not sufficient, and leaves the libguestfs VM running. Calling an explicit teardown on error in setup seems to fix the issue. I'm submitting an upstream bug and fix, and will backport it. upstream review in progress *** Bug 1022561 has been marked as a duplicate of this bug. *** verified on openstack-nova-compute-2013.2.3-5.el6ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2014-0578.html (In reply to Solly Ross from comment #14) > I have diagnosed the issue. It only occurs when you launch a VM with no > keypairs registered, and has to do with the libguestfs VFS driver. The > issue is caused by > nova/virt/disk/api:179 and nova/virt/disk/vfs/guestfs.py:136 -- namely, that > in case of an error in VFSGuestFS#setup(), VFSGuestFS#teardown() is not > called. Instead, `handle` is set to `None`, supposedly calling the implicit > teardown methods on `handle`. However, this is not sufficient, and leaves > the libguestfs VM running. > > Calling an explicit teardown on error in setup seems to fix the issue. > > I'm submitting an upstream bug and fix, and will backport it. Just found this. I am having the same issue openstack-nova-compute.2013.2.2.1.el6. To re-produce, just boot a VM and then delete it. There will be one qemu process left behind. Tried to use --key-name <registered keypair name> option but still the same thing. 2013.2.2-1 does not contain the backport of the fix. Please try 2013.2.4-1 instead, and see if you can still reproduce the issue. |
Created attachment 815465 [details] logs Description of problem: i am working witg glusterfs configured as cinder backend. when I run an instance I can see that there are 2 different qemu services launched when I delete the instance only one of the services is killed. Version-Release number of selected component (if applicable): openstack-nova-compute-2013.2-0.25.rc1.el6ost.noarch How reproducible: 100% Steps to Reproduce: 1. configure gluster as cinder backend 2. launch an instance from an image 3. run ps -elf |grep qemu 4. delete the instance 5. run ps -elf |grep qemu Actual results: there are 2 different qemu services run but only one is killed when we delete the instance Expected results: not sure two services should be run but if so, both should be killed when we delete the instance Additional info: [root@cougar07 ~(keystone_admin)]# nova list +--------------------------------------+-------+--------+------------+-------------+--------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------------+ | b2cda446-9ff2-405f-845a-422f5548738f | dafna | ACTIVE | None | Running | novanetwork=192.168.32.2 | +--------------------------------------+-------+--------+------------+-------------+--------------------------+ [root@cougar07 ~(keystone_admin)]# [root@cougar07 ~(keystone_admin)]# [root@cougar07 ~(keystone_admin)]# [root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu 2 S nova 28760 28700 10 80 0 - 212831 poll_s 18:30 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.28700 -initrd /var/tmp/.guestfs-162/initrd.28700 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.28700,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsNphmhH/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm 6 S qemu 29002 1 65 80 0 - 624337 futex_ 18:30 ? 00:00:09 /usr/libexec/qemu-kvm -name instance-0000004b -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b2cda446-9ff2-405f-845a-422f5548738f -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2-0.25.rc1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=b2cda446-9ff2-405f-845a-422f5548738f -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000004b.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:b0:3f:49,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.135:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 0 S root 29022 13613 0 80 0 - 25813 pipe_w 18:30 pts/2 00:00:00 grep qemu [root@cougar07 ~(keystone_admin)]# nova delete b2cda446-9ff2-405f-845a-422f5548738f [root@cougar07 ~(keystone_admin)]# nova list [root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu 2 S nova 28760 28700 3 80 0 - 212831 poll_s 18:30 ? 00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.28700 -initrd /var/tmp/.guestfs-162/initrd.28700 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/b2cda446-9ff2-405f-845a-422f5548738f/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.28700,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsNphmhH/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm 0 S root 29158 13613 0 80 0 - 25814 pipe_w 18:31 pts/2 00:00:00 grep qemu