Bug 1263998 - fail to shutdown win2008 guest in destination after migration
fail to shutdown win2008 guest in destination after migration
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Vadim Rozenfeld
Virtualization Bugs
Regression
: TestOnly
Depends On:
Blocks: 1401400 1288337
  Show dependency treegraph
 
Reported: 2015-09-17 05:41 EDT by jingzhao
Modified: 2017-02-09 04:39 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-02-09 04:39:08 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description jingzhao 2015-09-17 05:41:28 EDT
Description of problem:
Fail to shutdown win2008 r2 guest in destination after migration

Version-Release number of selected component (if applicable):
Guest win2008 R2
Host kernel:3.10.0-316.el7.x86_64
qemu-kvm-rhev-2.3.0-23.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot guest with cli in src:
/usr/libexec/qemu-kvm \
-M pc \
-cpu IvyBridge \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:55:60:41:0c:37,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-object iothread,id=iothread0 \
-drive file=/home/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1,iothread=iothread0 \
-object iothread,id=iothread1 \
-drive file=/home/win8r2/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0,format=raw \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=0 \
-drive file=/usr/share/virtio-win/virtio-win.iso,if=none,media=cdrom,id=drive-ide1,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide1,id=ide1 \
2.Boot vm with cli in destination
/usr/libexec/qemu-kvm \
-M pc \
-cpu IvyBridge \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,fds=20:21:22:23 20<>/dev/tap7 21<>/dev/tap7 22<>/dev/tap7 23<>/dev/tap7 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:55:60:41:0c:37,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-object iothread,id=iothread0 \
-drive file=/mnt/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1,iothread=iothread0 \
-object iothread,id=iothread1 \
-drive file=/home/win8r2/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0,format=raw \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=0 \
-drive file=/usr/share/virtio-win/virtio-win.iso,if=none,media=cdrom,id=drive-ide1,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide1,id=ide1 \
-incoming  tcp:0:5800 \

2. play media or cping files in the guest of src
3. migrate -d tcp:destip:5800 in source host 
4. In destination, shutdown the guest with "system_powerdown" in qemu-kvm

Actual results:
Guest of destination hang and failed to shutdown

Expected results:
Guest shutdown corretly and qemu-kvm exit corretly

Additional info:
Comment 1 jingzhao 2015-09-17 05:44:43 EDT
For steps4, modified as “In destination, shutdown the guest with "system_powerdown" in qemu-kvm after migrate completed”
Comment 3 Karen Noel 2015-09-17 06:42:56 EDT
Is this a regression from RHEL 7.1? 
Does this affect other Windows OS versions?
Does the guest shutdown properly if it is idle?
Can you shutdown the guest from the guest OS?
What is the guest doing when it is hung? Can you get a dump?
Comment 4 juzhang 2015-09-17 21:40:20 EDT
Hi Jing,

Could you reply comment3?

Best Regards,
Junyi
Comment 6 jingzhao 2015-09-18 03:09:17 EDT
I have reproduced this issue with following option on RHEL7.2
  - delete network device and dataplane
1. boot guest in src with cli;
/usr/libexec/qemu-kvm \
-M pc \
-cpu IvyBridge \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-drive file=/home/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1 \
-nodefaults
2.Boot vm with -incoming tcp:0:5800 in destination
3.migrate in the src host
4.shutdown from guest OS or "system_powerdown" after migration

The same issue can be reproduced.
Comment 7 jingzhao 2015-09-18 04:16:11 EDT
Reproduced this issue with option(delete dataplane, keep network) on RHEL7.2
1. boot guest in src with cli;
-M pc \
-cpu SandyBridge \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=42:55:4a:69:c6:e3,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-drive file=/home/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1 \

2.Boot vm with -incoming tcp:0:5800 in destination
3.migrate in the src host
4.shutdown from guest OS or "system_powerdown" after migration
Comment 8 jingzhao 2015-09-18 04:23:54 EDT
Verified with 7.1 and the issue didn't reproduced.
Guest: win2008 r2
Host kernel: 3.10.0-229.el7.x86_64
qemu-kvm-rhev-2.1.2-23.el7.x86_64.rpm
1. deleted dataplane and keep network, boot cli with follow cli:
1. boot guest in src with cli;
-M pc \
-cpu SandyBridge \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=42:55:4a:69:c6:e3,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-drive file=/home/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1 \

2. Boot vm with -incoming tcp:0:5800
3. Copying some files 
4. migrated in the src host
5. Shutdown from guest or "system_powerdown" after migration 
---Guest OS shutdown successfully and qemu exit

Also verified it with delete network device option, guest also can be shutdown through guestOS shutdown or "system_powerdown"
Comment 11 Vadim Rozenfeld 2015-10-05 05:28:51 EDT
Is the problem reproducible with ide device instead of virtio block device?
Can we try enabling NMICrashDump ( https://technet.microsoft.com/en-us/library/cc783271%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396 )
generating a crash dump file with NMI qemu command after the system is hung?

Thanks,
Vadim.
Comment 17 juzhang 2016-12-15 05:53:56 EST
Hi Jin,

Could you reply comment16?

Best Regards,
Junyi
Comment 18 jingzhao 2016-12-16 01:58:37 EST
Tested it and didn't reproduce the issue on 7.3.z version

[root@hp-z800-01 home]# uname -r
3.10.0-514.6.1.el7.x86_64
[root@hp-z800-01 home]# rpm -qa |grep qemu-kvm-rhev
qemu-kvm-rhev-2.6.0-28.el7_3.2.x86_64

Test scenario
1. Boot guest with command [1]
2. copy large files in guest
3. migrate during copying step
4. "system_powerdown" in hmp after migrate finished.

BTW: also test it with following test scenario 
a. without network and dataplane
b. with network but without dataplane

[1]
/usr/libexec/qemu-kvm \
-M pc \
-cpu Nehalem \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:55:60:41:0c:37,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-object iothread,id=iothread0 \
-drive file=/mnt/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1,iothread=iothread0 \
-object iothread,id=iothread1 \
-drive file=/mnt/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0,format=raw \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=0 \
-drive file=/usr/share/virtio-win/virtio-win-1.9.0.iso,if=none,media=cdrom,id=drive-ide1,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide1,id=ide1 \


Thanks
Jing Zhao
Comment 20 Vadim Rozenfeld 2017-02-08 22:02:48 EST
Can we please check if it works on rhel 7.4?

Thanks,
Vadim.
Comment 21 juzhang 2017-02-08 23:16:30 EST
(In reply to Vadim Rozenfeld from comment #20)
> Can we please check if it works on rhel 7.4?
> 
> Thanks,
> Vadim.

Hi Jing,

Could you have a try by using latest qemu-kvm-rhev? 

Best Regards,
Junyi
Comment 22 jingzhao 2017-02-09 03:00:00 EST
Tested it with following version and didn't reproduce the bz

Host:
  kernel-3.10.0-552.el7.x86_64
  qemu-kvm-rhev-2.8.0-3.el7.x86_64

Guest:
 en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso

Test Steps:

  1. Boot guest with qemu command [1]
  2. copy large files in guest during migration
  3. "system_powerdown" on hmp after migration 

[1]
/usr/libexec/qemu-kvm \
-M pc \
-cpu IvyBridge \
-nodefaults -rtc base=utc -no-hpet \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel7.4 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:55:60:41:0c:37,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-object iothread,id=iothread0 \
-drive file=/mnt/test/bz1263998/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1,iothread=iothread0 \
-object iothread,id=iothread1 \
-drive file=/mnt/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0,format=raw \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=0 \
-drive file=/usr/share/virtio-win/virtio-win-1.9.0.iso,if=none,media=cdrom,id=drive-ide1,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide1,id=ide1 \


Thanks
Jing Zhao

Note You need to log in before you can comment on or make changes to this bug.