677728 – 2.2.z: after migration backend calls destroy for VM but VM still appears as Down on source host

Bug 677728 - 2.2.z: after migration backend calls destroy for VM but VM still appears as Down on source host

Summary: 2.2.z: after migration backend calls destroy for VM but VM still appears as D...

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat Enterprise Linux 5
Classification:	Red Hat
Component:	vdsm22
Sub Component:
Version:	5.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Igor Lvovsky
QA Contact:	yeylon@redhat.com
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	684586 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2011-02-15 17:13 UTC by Dafna Ron
Modified:	2016-04-18 06:38 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2011-06-20 09:00:18 UTC
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)
logs (722.77 KB, application/x-gzip) 2011-02-15 17:13 UTC, Dafna Ron	no flags	Details
new vdsm log (5.10 MB, text/x-log) 2011-03-24 08:27 UTC, Igor Lvovsky	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Dafna Ron 2011-02-15 17:13:01 UTC

Created attachment 478925 [details]
logs

Description of problem:

when running migrateble VM's and non-migratble VM's on the same host -> creating storage disconnection on host, the VM's which can be migrated are migrated but remain in source host in down state even though backend sends destroy.  


Version-Release number of selected component (if applicable):
ic92
vdsm22-debuginfo-4.5-63.17.el5_6
vdsm22-cli-4.5-63.17.el5_6
vdsm22-faqemu-4.5-63.17.el5_6
vdsm22-4.5-63.17.el5_6
kvm-qemu-img-83-224.el5
vdsm22-faqemu-4.5-63.17.el5_6


How reproducible:
100%

Steps to Reproduce:
1. run both pin-to-host and non-pinned VM's on source host
2. block storage using iptables
3. VM will migrate to destination host

  
Actual results:

VM that migrated are shown on source host in down state and in destination host in up state


Expected results:

VM's should only appear as up on destination after backend send destroy. 

Additional info:
complete logs attached

[root@blond-vdsf ~]# less /var/log/vdsm/vdsm.log |grep '13daec20-4dd0-43ce-95f0-6f1d3ddc22ac.*destroy'  
Thread-167::DEBUG::2011-02-15 18:22:14,653::vm::1309::vds.vmlog.13daec20-4dd0-43ce-95f0-6f1d3ddc22ac::destroy Called
Thread-167::DEBUG::2011-02-15 18:22:15,816::vm::1361::vds.vmlog.13daec20-4dd0-43ce-95f0-6f1d3ddc22ac::Total desktops after destroy of 13daec20-4dd0-43ce-95f0-6f1d3ddc22ac is 6
Thread-79::DEBUG::2011-02-15 18:41:08,529::vm::1309::vds.vmlog.13daec20-4dd0-43ce-95f0-6f1d3ddc22ac::destroy Called
Thread-79::DEBUG::2011-02-15 18:41:09,693::vm::1361::vds.vmlog.13daec20-4dd0-43ce-95f0-6f1d3ddc22ac::Total desktops after destroy of 13daec20-4dd0-43ce-95f0-6f1d3ddc22ac is 2




[root@blond-vdsf tmp]# vdsClient -s 0 list table |grep -i down
13daec20-4dd0-43ce-95f0-6f1d3ddc22ac  23216  RHEL6                Down                                     
493980b6-730a-4ba9-babf-521482977559  23542  SesyPrep2003tmp      Down                                     
d0199102-4a4b-42b4-b51f-d6fc5c1e714a  23330  2008R2               Down                                     
8572858d-6874-44c4-9de0-f352bc18ba5d  23439  2008                 Down         



[root@blond-vdsg tmp]# vdsClient -s 0 list table
493980b6-730a-4ba9-babf-521482977559   4572  SesyPrep2003tmp      Up                                       
8572858d-6874-44c4-9de0-f352bc18ba5d   4458  2008                 Up                                       
13daec20-4dd0-43ce-95f0-6f1d3ddc22ac   4237  RHEL6                Up                                       
d0199102-4a4b-42b4-b51f-d6fc5c1e714a   4347  2008R2               Up

Comment 1 Dan Kenigsberg 2011-02-17 20:37:58 UTC

This seems similar to RHEV 2.3's bug 672493: we fail destroy() since tearDown cannot be accomplished without active storage. The only problem is that RHEV-M does not expect destroy() to fail; we may need to move such VM to "Powering down" instead.

I'm not sure that we want such an intrusive change in zsteam, but Igor, would you take a look?

Comment 2 Dan Kenigsberg 2011-03-14 09:07:38 UTC

*** Bug 684586 has been marked as a duplicate of this bug. ***

Comment 3 Igor Lvovsky 2011-03-22 13:12:35 UTC

 Dafna,
I need to check whether 'qemu' process of this VM still exists on source host

Comment 4 Dafna Ron 2011-03-23 12:49:50 UTC

this is source host: no longer SPM and non-operational. 

[root@blond-vdsf ~]# vdsClient -s 0 list table
2c082f3c-bb05-49b4-8d33-e8b97360ea00   5492  PinToHost1           Paused                                   
b6b1c807-10fb-4401-8b03-43bdec530e95   5921  MigrateVM1           Down                                     
92e827f4-dd2b-4004-8efc-faa91a6f5090   5629  PinToHost2           Paused                                   
f7f57ba3-b077-4be2-9938-b7b9864f68ac   5776  MigrateVM2           Down                                     
[root@blond-vdsf ~]# ps -elf |grep qemu
0 S vdsm      5483     1  0  76  -5 - 15963 wait   14:36 ?        00:00:00 /bin/sh -c /usr/bin/sudo /usr/bin/tunctl -d rtl8139_10_1; /usr/bin/sudo /usr/bin/tunctl -b -u vdsm -t rtl8139_10_1;/usr/bin/sudo /sbin/ip link set dev rtl8139_10_1 address fe:1a:4a:23:71:00;/usr/bin/sudo /sbin/ip link set dev rtl8139_10_1 up;/usr/bin/sudo /usr/sbin/brctl addif rhevm rtl8139_10_1;TZ=UTC /bin/nice -n 5 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack  -startdate 2011-03-23T12:36:08  -name PinToHost1  -smp 1,cores=1  -k en-us  -m 1024  -boot c  -net nic,vlan=1,macaddr=00:1a:4a:23:71:00,model=rtl8139 -net tap,vlan=1,ifname=rtl8139_10_1,script=no  -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/eda275b7-b740-4ccb-91d7-8d3a7f5c57b0/23129365-2c8e-45ed-b56a-10d876ea74b5,media=disk,if=ide,cache=off,index=0,serial=cb-91d7-8d3a7f5c57b0,boot=off,format=qcow2,werror=stop -pidfile /var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.pid -vnc 0:10,password  -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm  -M rhel5.5.0  -notify all  -balloon none  -smbios type=1,manufacturer="Red Hat",product="RHEL",version=5Server-5.6.0.3,serial="44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca",uuid="2c082f3c-bb05-49b4-8d33-e8b97360ea00"  -vmchannel di:0200,unix:/var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.guest.socket,server -monitor unix:/var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.monitor.socket,server 1>/var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.stdio.dump 2>&1; /usr/bin/sudo /usr/bin/tunctl -d rtl8139_10_1; 
0 S vdsm      5492  5483  4  75   0 - 312427 -     14:36 ?        00:00:19 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet -rtc-td-hack -startdate 2011-03-23T12:36:08 -name PinToHost1 -smp 1,cores=1 -k en-us -m 1024 -boot c -net nic,vlan=1,macaddr=00:1a:4a:23:71:00,model=rtl8139 -net tap,vlan=1,ifname=rtl8139_10_1,script=no -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/eda275b7-b740-4ccb-91d7-8d3a7f5c57b0/23129365-2c8e-45ed-b56a-10d876ea74b5,media=disk,if=ide,cache=off,index=0,serial=cb-91d7-8d3a7f5c57b0,boot=off,format=qcow2,werror=stop -pidfile /var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.pid -vnc 0:10,password -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm -M rhel5.5.0 -notify all -balloon none -smbios type=1,manufacturer=Red Hat,product=RHEL,version=5Server-5.6.0.3,serial=44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca,uuid=2c082f3c-bb05-49b4-8d33-e8b97360ea00 -vmchannel di:0200,unix:/var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.guest.socket,server -monitor unix:/var/vdsm/2c082f3c-bb05-49b4-8d33-e8b97360ea00.monitor.socket,server
0 S vdsm      5620     1  0  73  -5 - 15963 wait   14:36 ?        00:00:00 /bin/sh -c /usr/bin/sudo /usr/bin/tunctl -d rtl8139_11_1; /usr/bin/sudo /usr/bin/tunctl -b -u vdsm -t rtl8139_11_1;/usr/bin/sudo /sbin/ip link set dev rtl8139_11_1 address fe:1a:4a:23:71:01;/usr/bin/sudo /sbin/ip link set dev rtl8139_11_1 up;/usr/bin/sudo /usr/sbin/brctl addif rhevm rtl8139_11_1;TZ=UTC /bin/nice -n 5 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack  -startdate 2011-03-23T13:36:09  -name PinToHost2  -smp 1,cores=1  -k en-us  -m 1024  -boot c  -net nic,vlan=1,macaddr=00:1a:4a:23:71:01,model=rtl8139 -net tap,vlan=1,ifname=rtl8139_11_1,script=no  -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/a8c1554f-c397-4efc-b834-a3c62e8081b6/e6d19e3f-d9d1-4764-adaf-7af631655064,media=disk,if=ide,cache=off,index=0,serial=fc-b834-a3c62e8081b6,boot=off,format=qcow2,werror=stop -pidfile /var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.pid -vnc 0:11,password  -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm  -M rhel5.5.0  -notify all  -balloon none  -smbios type=1,manufacturer="Red Hat",product="RHEL",version=5Server-5.6.0.3,serial="44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca",uuid="92e827f4-dd2b-4004-8efc-faa91a6f5090"  -vmchannel di:0200,unix:/var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.guest.socket,server -monitor unix:/var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.monitor.socket,server 1>/var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.stdio.dump 2>&1; /usr/bin/sudo /usr/bin/tunctl -d rtl8139_11_1; 
0 S vdsm      5629  5620 22  75   0 - 312427 -     14:36 ?        00:01:40 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet -rtc-td-hack -startdate 2011-03-23T13:36:09 -name PinToHost2 -smp 1,cores=1 -k en-us -m 1024 -boot c -net nic,vlan=1,macaddr=00:1a:4a:23:71:01,model=rtl8139 -net tap,vlan=1,ifname=rtl8139_11_1,script=no -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/a8c1554f-c397-4efc-b834-a3c62e8081b6/e6d19e3f-d9d1-4764-adaf-7af631655064,media=disk,if=ide,cache=off,index=0,serial=fc-b834-a3c62e8081b6,boot=off,format=qcow2,werror=stop -pidfile /var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.pid -vnc 0:11,password -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm -M rhel5.5.0 -notify all -balloon none -smbios type=1,manufacturer=Red Hat,product=RHEL,version=5Server-5.6.0.3,serial=44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca,uuid=92e827f4-dd2b-4004-8efc-faa91a6f5090 -vmchannel di:0200,unix:/var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.guest.socket,server -monitor unix:/var/vdsm/92e827f4-dd2b-4004-8efc-faa91a6f5090.monitor.socket,server
0 S vdsm      5767     1  0  76  -5 - 15963 wait   14:36 ?        00:00:00 /bin/sh -c /usr/bin/sudo /usr/bin/tunctl -d virtio_12_1; /usr/bin/sudo /usr/bin/tunctl -b -u vdsm -t virtio_12_1;/usr/bin/sudo /sbin/ip link set dev virtio_12_1 address fe:1a:4a:23:71:03;/usr/bin/sudo /sbin/ip link set dev virtio_12_1 up;/usr/bin/sudo /usr/sbin/brctl addif rhevm virtio_12_1;TZ=UTC /bin/nice -n 5 /usr/libexec/qemu-kvm -no-hpet -no-kvm-pit-reinjection  -usbdevice tablet  -rtc-td-hack  -startdate 2011-03-23T12:36:12  -name MigrateVM2  -smp 1,cores=1  -k en-us  -m 1024  -boot c  -net nic,vlan=1,macaddr=00:1a:4a:23:71:03,model=virtio -net tap,vlan=1,ifname=virtio_12_1,script=no  -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/4ef9001c-23d8-48b9-b586-3f0a6c41adbd/5df5122d-c0b8-4c68-b3dc-6b6d655135de,media=disk,if=virtio,cache=off,serial=b9-b586-3f0a6c41adbd,boot=on,format=qcow2,werror=stop -pidfile /var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.pid -vnc 0:12,password  -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm  -M rhel5.5.0  -notify all  -balloon none  -smbios type=1,manufacturer="Red Hat",product="RHEL",version=5Server-5.6.0.3,serial="44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca",uuid="f7f57ba3-b077-4be2-9938-b7b9864f68ac"  -vmchannel di:0200,unix:/var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.guest.socket,server -monitor unix:/var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.monitor.socket,server 1>/var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.stdio.dump 2>&1; /usr/bin/sudo /usr/bin/tunctl -d virtio_12_1; 
0 S vdsm      5776  5767  6  75   0 - 318620 -     14:36 ?        00:00:30 /usr/libexec/qemu-kvm -no-hpet -no-kvm-pit-reinjection -usbdevice tablet -rtc-td-hack -startdate 2011-03-23T12:36:12 -name MigrateVM2 -smp 1,cores=1 -k en-us -m 1024 -boot c -net nic,vlan=1,macaddr=00:1a:4a:23:71:03,model=virtio -net tap,vlan=1,ifname=virtio_12_1,script=no -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/4ef9001c-23d8-48b9-b586-3f0a6c41adbd/5df5122d-c0b8-4c68-b3dc-6b6d655135de,media=disk,if=virtio,cache=off,serial=b9-b586-3f0a6c41adbd,boot=on,format=qcow2,werror=stop -pidfile /var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.pid -vnc 0:12,password -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm -M rhel5.5.0 -notify all -balloon none -smbios type=1,manufacturer=Red Hat,product=RHEL,version=5Server-5.6.0.3,serial=44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca,uuid=f7f57ba3-b077-4be2-9938-b7b9864f68ac -vmchannel di:0200,unix:/var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.guest.socket,server -monitor unix:/var/vdsm/f7f57ba3-b077-4be2-9938-b7b9864f68ac.monitor.socket,server
0 S vdsm      5912     1  0  79  -5 - 15963 wait   14:36 ?        00:00:00 /bin/sh -c /usr/bin/sudo /usr/bin/tunctl -d virtio_13_1; /usr/bin/sudo /usr/bin/tunctl -b -u vdsm -t virtio_13_1;/usr/bin/sudo /sbin/ip link set dev virtio_13_1 address fe:1a:4a:23:71:02;/usr/bin/sudo /sbin/ip link set dev virtio_13_1 up;/usr/bin/sudo /usr/sbin/brctl addif rhevm virtio_13_1;TZ=UTC /bin/nice -n 5 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet  -rtc-td-hack  -startdate 2011-03-23T12:36:14  -name MigrateVM1  -smp 1,cores=1  -k en-us  -m 1024  -boot c  -net nic,vlan=1,macaddr=00:1a:4a:23:71:02,model=virtio -net tap,vlan=1,ifname=virtio_13_1,script=no  -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/665c742e-dffa-4ee6-b0f2-dfc4964295d9/7c2c6b53-923d-405f-b96b-a6e336afa813,media=disk,if=virtio,cache=off,serial=e6-b0f2-dfc4964295d9,boot=on,format=qcow2,werror=stop -pidfile /var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.pid -vnc 0:13,password  -incoming tcp::49153 -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm  -M rhel5.5.0  -notify all  -balloon none  -smbios type=1,manufacturer="Red Hat",product="RHEL",version=5Server-5.6.0.3,serial="44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca",uuid="b6b1c807-10fb-4401-8b03-43bdec530e95"  -vmchannel di:0200,unix:/var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.guest.socket,server -monitor unix:/var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.monitor.socket,server 1>/var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.stdio.dump 2>&1; /usr/bin/sudo /usr/bin/tunctl -d virtio_13_1; 
0 S vdsm      5921  5912  5  75   0 - 319291 -     14:36 ?        00:00:26 /usr/libexec/qemu-kvm -no-hpet -usbdevice tablet -rtc-td-hack -startdate 2011-03-23T12:36:14 -name MigrateVM1 -smp 1,cores=1 -k en-us -m 1024 -boot c -net nic,vlan=1,macaddr=00:1a:4a:23:71:02,model=virtio -net tap,vlan=1,ifname=virtio_13_1,script=no -drive file=/rhev/data-center/26667093-cfe4-46ea-8ec4-43b4150c796a/a7536651-1aa6-4ff4-a71e-a1d570475fb7/images/665c742e-dffa-4ee6-b0f2-dfc4964295d9/7c2c6b53-923d-405f-b96b-a6e336afa813,media=disk,if=virtio,cache=off,serial=e6-b0f2-dfc4964295d9,boot=on,format=qcow2,werror=stop -pidfile /var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.pid -vnc 0:13,password -incoming tcp::49153 -cpu qemu64,+sse2,+cx16,+sse4a,+misalignsse,+popcnt,+abm -M rhel5.5.0 -notify all -balloon none -smbios type=1,manufacturer=Red Hat,product=RHEL,version=5Server-5.6.0.3,serial=44454C4C-3800-1046-8035-B1C04F34344A_00:22:19:14:7e:ca,uuid=b6b1c807-10fb-4401-8b03-43bdec530e95 -vmchannel di:0200,unix:/var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.guest.socket,server -monitor unix:/var/vdsm/b6b1c807-10fb-4401-8b03-43bdec530e95.monitor.socket,server
0 S root     10502  4234  0  77   0 - 15298 pipe_w 14:43 pts/0    00:00:00 grep qemu
[root@blond-vdsf ~]#

Comment 5 Igor Lvovsky 2011-03-24 08:27:25 UTC

Created attachment 487249 [details]
new vdsm log

Note You need to log in before you can comment on or make changes to this bug.