Bug 1022561 - nova: killing qemu pid which remains after delete of instances will cause openstack-nova-compute to die
nova: killing qemu pid which remains after delete of instances will cause ope...
Status: CLOSED DUPLICATE of bug 1022627
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
4.0
x86_64 Linux
unspecified Severity medium
: ---
: 4.0
Assigned To: Solly Ross
Ami Jeain
storage
: Reopened, Unconfirmed, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-23 10:16 EDT by Dafna Ron
Modified: 2014-04-07 11:02 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-04-07 11:02:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log (11.10 MB, application/x-xz)
2013-10-23 10:16 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1243718 None None None Never
OpenStack gerrit 67586 None None None Never

  None (edit)
Description Dafna Ron 2013-10-23 10:16:18 EDT
Created attachment 815437 [details]
log

Description of problem:

I am working with gluster as cinder backend and when I boot an instance I can see two different qemu pid's run: 

[root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu
2 S nova     13775 13645  6  80   0 - 212831 poll_s 16:51 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.13645 -initrd /var/tmp/.guestfs-162/initrd.13645 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/2f469c9d-2bf8-44ba-ac9a-e280289802bf/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.13645,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsWTO2Jc/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm
6 S qemu     14018     1 72  80   0 - 217865 poll_s 16:51 ?       00:00:19 /usr/libexec/qemu-kvm -name instance-00000035 -S -M rhel6.5.0 -cpu Opteron_G3,+nodeid_msr,+wdt,+skinit,+ibs,+osvw,+3dnowprefetch,+cr8legacy,+extapic,+cmp_legacy,+3dnow,+3dnowext,+pdpe1gb,+fxsr_opt,+mmxext,+ht,+vme -enable-kvm -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 2f469c9d-2bf8-44ba-ac9a-e280289802bf -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.2-0.25.rc1.el6ost,serial=44454c4c-4a00-1044-804c-b5c04f39354a,uuid=2f469c9d-2bf8-44ba-ac9a-e280289802bf -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000035.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/2f469c9d-2bf8-44ba-ac9a-e280289802bf/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:37:43:23,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/2f469c9d-2bf8-44ba-ac9a-e280289802bf/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 10.35.160.135:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

when I delete the instance I can see that only one of these pid's are left 

[root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu
2 S nova     13775 13645  4  80   0 - 212831 poll_s 16:51 ?       00:00:02 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.13645 -initrd /var/tmp/.guestfs-162/initrd.13645 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/2f469c9d-2bf8-44ba-ac9a-e280289802bf/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.13645,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsWTO2Jc/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm
0 S root     14135 13613  0  80   0 - 25813 pipe_w 16:52 pts/2    00:00:00 grep qemu

if I kill it nova-compute will die: 

[root@cougar07 ~(keystone_admin)]# kill -9 13645
[root@cougar07 ~(keystone_admin)]# 
[root@cougar07 ~(keystone_admin)]# 
[root@cougar07 ~(keystone_admin)]# ps -elf |grep qemu
0 S root     14140 13613  0  80   0 - 25813 pipe_w 16:52 pts/2    00:00:00 grep qemu
[root@cougar07 ~(keystone_admin)]# 
[root@cougar07 ~(keystone_admin)]# /etc/init.d/openstack-nova-compute status
openstack-nova-compute dead but pid file exists
[root@cougar07 ~(keystone_admin)]# 

Version-Release number of selected component (if applicable):

openstack-nova-compute-2013.2-0.25.rc1.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. configure gluster as cinder backend
2. boot an instance and run ps on qemu 
3. delete instance 
4. kill the left over pid

Actual results:

nova-compute dies as well

Expected results:

nova should not die

Additional info:
Comment 1 Solly Ross 2013-10-31 11:44:30 EDT
Can you get me logs from the other compute services (namely scheduler)?  Additionally, what is your setup (I want to make sure I duplicate it correctly) -- what nodes, etc.  Also, does this only happen when running Cinder with a Gluster backend?
Comment 2 Dafna Ron 2013-10-31 11:56:26 EDT
will give setup privately (logs are there). 
I am not sure if this is gluster related since I only have a gluster setup at this time.
Comment 3 Solly Ross 2013-10-31 15:43:18 EDT
Notes: processes are left in STAT=S (interruptible sleep) on account of poll_schedule_timeout (i.e. they are waiting for a poll call).
Comment 4 Solly Ross 2013-11-01 12:27:29 EDT
This makes me suspect that the cause is the Gluster driver.  Will do some more digging.
Comment 6 Solly Ross 2013-12-19 15:15:56 EST
could not reproduce this anymore
Comment 7 Dafna Ron 2014-02-26 09:07:38 EST
I'm re-openeing since this is 100% reproduced on my setup (4.0 last puddle release)
If you like to investigate further and cannot reproduce on your setup please contact me and I will give you access to my setup. 

root@puma31 ~]# ps -elf |grep qemu
0 S root      9993  9708  0  80   0 - 25813 pipe_w 16:05 pts/0    00:00:00 grep qemu
2 Z nova     16552  8454  0  80   0 -     0 exit   Feb21 ?        00:03:03 [qemu-kvm] <defunct>
2 S nova     18210  8454  0  80   0 - 212852 poll_s Feb21 ?       00:02:54 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.8454 -initrd /var/tmp/.guestfs-162/initrd.8454 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/c1d544cb-1d1d-435f-9672-af5347d29c0e/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.8454,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsRV7oiz/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm
[root@puma31 ~]# ps -elf |grep qemu
0 S root      9996  9708  0  80   0 - 25813 pipe_w 16:05 pts/0    00:00:00 grep qemu
2 Z nova     16552  8454  0  80   0 -     0 exit   Feb21 ?        00:03:03 [qemu-kvm] <defunct>
2 S nova     18210  8454  0  80   0 - 212852 poll_s Feb21 ?       00:02:54 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-162/kernel.8454 -initrd /var/tmp/.guestfs-162/initrd.8454 -device virtio-scsi-pci,id=scsi -drive file=/var/lib/nova/instances/c1d544cb-1d1d-435f-9672-af5347d29c0e/disk,cache=none,format=qcow2,id=hd0,if=none -device scsi-hd,drive=hd0 -drive file=/var/tmp/.guestfs-162/root.8454,snapshot=on,id=appliance,if=none,cache=unsafe -device scsi-hd,drive=appliance -device virtio-serial -serial stdio -device sga -chardev socket,path=/tmp/libguestfsRV7oiz/guestfsd.sock,id=channel0 -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 -append panic=1 console=ttyS0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=xterm
[root@puma31 ~]# 
[root@puma31 ~]# 
[root@puma31 ~]# 
[root@puma31 ~]# 
[root@puma31 ~]# kill -9 8454
[root@puma31 ~]# /etc/init.d/openstack-nova-compute status
openstack-nova-compute dead but pid file exists
[root@puma31 ~]#
Comment 11 Solly Ross 2014-02-27 16:54:51 EST
@Dafna: is this related to https://bugzilla.redhat.com/show_bug.cgi?id=1022627 ?  Did you try the proposed fix for that?  I believe the two are related.  Please try the fix proposed upstream for the bug.
Comment 13 Dave Allan 2014-03-06 11:35:06 EST
Solly, can you provide Dafna a scratch build with the proposed fix, or work with her to patch her systems for her so she can test?
Comment 14 Solly Ross 2014-03-18 13:58:23 EDT
I've tested the patch on the QE systems, and it appears that it works.  The backport is being checked upstream, so it should be in next time someone rebases RHOS 4.0.z off of stable/havana upstream.
Comment 17 Xavier Queralt 2014-04-07 05:44:24 EDT
I think this could be closed as a duplicate of bug 1022627

What do you think Solly?
Comment 18 Solly Ross 2014-04-07 11:02:53 EDT
@Xavier Queralt: yeah, sounds good.

*** This bug has been marked as a duplicate of bug 1022627 ***

Note You need to log in before you can comment on or make changes to this bug.