Bug 1037480

Summary: Failed to do S4 after migration from a RHEL6.5 host to RHEL7.0 host
Product: Red Hat Enterprise Linux 7 Reporter: Qunfang Zhang <qzhang>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: amit.shah, dyuan, hhuang, huding, juzhang, lersek, michen, qzhang, rbalakri, rkrcmar, virt-maint, zhwang, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-02 10:04:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 923626    
Attachments:
Description Flags
dmesg-vmcore.txt file (-M rhel6.4.0)
none
vmcore.txt (first time s4 after migration) with comment 7 build
none
vmcore.txt (second time s4 after migration) with comment 7 build
none
the screenshot that win7sp1-64 guest cannot resume after S4 none

Description Qunfang Zhang 2013-12-03 09:27:38 UTC
Description of problem:
Migrate a guest from RHEL6.5 released host to RHEL7.0 host, and then do pm-hibernate, it's failed. If I boot up guest with same command line on RHEL7.0 host and then do S4, succeed. 

Version-Release number of selected component (if applicable):
kernel-3.10.0-57.el7.x86_64
qemu-kvm-1.5.3-20.el7.x86_64

Guest: 
RHEL7.0

How reproducible:
Always

Steps to Reproduce:
1. Boot up a guest on RHEL6.5 host:

(gdb) r -cpu SandyBridge -M rhel6.5.0 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -name rhel7.0-64 -uuid 9a0e67ec-f286-d8e7-0548-0c1c9ec93009 -nodefconfig -nodefaults -monitor stdio -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -drive file=gluster://10.66.4.113/gv0/rhel7.0-64-qzhang.raw,if=none,id=drive-disk0,format=raw,cache=none,id=disk0 -device scsi-hd,drive=drive-disk0,bus=scsi0.0,id=scsi-disk0,bootindex=1  -drive file=gluster://10.66.4.113/gv0/boot.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device scsi-cd,drive=drive-ide0-1-0,bus=scsi0.0,id=scsi-cdrom -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:8a,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=port1,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=port2,bus=virtio-serial0.0,id=port2 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -vnc :10 -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6  -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -drive file=gluster://10.66.4.113/gv0/disk1.qcow2,if=none,id=disk1,format=qcow2,cache=none,id=disk1 -device virtio-blk-pci,bus=pci.0,addr=0x9,drive=disk1,id=disk1 -drive file=gluster://10.66.4.113/gv0/disk2.qcow2,if=none,id=disk2,format=qcow2,cache=none,id=disk2 -device ide-drive,bus=ide.0,unit=1,drive=disk2,id=disk2  -netdev tap,id=hostnet1,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device e1000,netdev=hostnet1,id=net1,mac=52:54:00:d5:51:2a,bus=pci.0,addr=0xa -netdev tap,id=hostnet2,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device e1000,netdev=hostnet2,id=net2,mac=52:54:00:d5:51:3a,bus=pci.0,addr=0xb 

2. Boot up the guest on destination (rhel7.0) host with "-incoming tcp:0:5800"

3. Migrate the guest to 7.0 host.

4. After migration, do S4 inside guest.
# cat /sys/power/state 
freeze mem disk
#pm-hiberate

Actual results:
Guest hangs during hibernating, failed to do S4.

Expected results:
Guest should hibernate successfully.

Additional info:

Top info on destination host during guest S4:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                      
 7731 root      20   0 6097116 1.143g   7548 S 100.9 15.5   0:37.99 qemu-kvm                                     
 7460 root      20   0  123660   1596   1124 R   0.3  0.0   0:03.00 top

Comment 2 Qunfang Zhang 2013-12-03 09:56:26 UTC
Created attachment 831955 [details]
dmesg-vmcore.txt file (-M rhel6.4.0)

When migrate guest with "-M rhel6.4.0" with same steps, guest core dump during S4 after migration.

Comment 3 Qunfang Zhang 2013-12-04 07:52:27 UTC
Retest the following scenarios suggested by Amit:

(1) with -M rhel6.5.0 machine type:
Actually comment 0 is using "-M rhel6.5.0" already. So, still hit the problem.

(2) Without virtio device.
Still failed with same phenomenon.

(3)Test a RHEL6.5-64 guest (comment 0 is using a rhel7.0 guest):
With and without virtio device: failed,same phenomenon.

"top" shows 100% cpu consumption, and I can not get any log from the isa serial console.

Comment 4 Amit Shah 2014-01-10 05:25:21 UTC
Orit, didn't you look at something similar?  There was one where the firmware sizes are different across 6.5 and 7.0, but this doesn't seem to be that (firmware is migrated too).  Anything that strikes you about this?

Comment 5 Orit Wasserman 2014-01-13 07:52:24 UTC
(In reply to Amit Shah from comment #4)
> Orit, didn't you look at something similar?  There was one where the
> firmware sizes are different across 6.5 and 7.0, but this doesn't seem to be
> that (firmware is migrated too).  Anything that strikes you about this?

We have problems with restarting guest after migration from 6.5 to 7.0, could be related but I'm not sure.

Comment 9 Qunfang Zhang 2014-04-16 07:29:02 UTC
Created attachment 886768 [details]
vmcore.txt (first time s4 after migration) with comment 7 build

Comment 10 Qunfang Zhang 2014-04-16 07:29:48 UTC
Created attachment 886769 [details]
vmcore.txt (second time s4 after migration) with comment 7 build

Comment 12 huiqingding 2014-05-08 07:50:40 UTC
Created attachment 893546 [details]
the screenshot that win7sp1-64 guest cannot resume after S4

I migrate a win7sp1-64 guest from RHEL6.6 host to RHEL7.0 host, do S4 and then start guest on dest host, the guest cannot resume. The screenshot is as the attachment file.

RHEL6.6 host:
kernel-2.6.32-459.el6.x86_64
qemu-kvm-0.12.1.2-2.425.el6.x86_64

RHEL7.0 host:
kernel-3.10.0-123.el7.x86_64
qemu-kvm-1.5.3-60.el7_0.2.x86_64

qxl driver of win7sp1-64 guest is qxl-win-0.1-21.

Don't do migration, on RHEL6.6 host, boot win7sp1-64 guest and do S4. the guest can resume successfully. On RHEL7.0 host, boot win7sp1-64 guest and do S4. the guest can resume successfully.

The command line is as following:
/usr/libexec/qemu-kvm -M rhel6.5.0 -cpu SandyBridge,hv_relaxed  -enable-kvm  -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 -numa node,cpus=1 -numa node,cpus=2 -numa node,cpus=3 -nodefconfig -nodefaults -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -global ide-drive.physical_block_size=4096 -global ide-drive.logical_block_size=4096 -global virtio-blk-pci.physical_block_size=512 -global virtio-blk-pci.logical_block_size=512 -boot order=cdn,once=n,menu=on,strict=on,reboot-timeout=60000 -k en-us -soundhw ac97 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x5,indirect_desc=on,event_idx=on,multifunction=on,rombar=100 -monitor stdio -name test-all-qemu-kvm-option -uuid `uuidgen`  -drive file=/mnt/virtio-blk-disk,if=none,id=drive-virtio-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,media=disk,snapshot=off,bus=1,unit=1 -device virtio-blk-pci,scsi=off,drive=drive-virtio-disk,id=virtio-disk,bus=pci.0,addr=0x7,physical_block_size=512,logical_block_size=512,multifunction=on,scsi=on,event_idx=on,indirect_desc=on,vectors=32,x-data-plane=off,ioeventfd=on,serial=fuxc,discard_granularity=1,min_io_size=4096,opt_io_size=4096  -usbdevice tablet -usbdevice mouse  -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,ifname=fuxc-net -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:03:04:05,bus=pci.0,addr=0x9,multifunction=on,status=on,gso=on,ioeventfd=on,vectors=8,indirect_desc=off,event_idx=off,guest_tso4=off,guest_tso6=on,guest_ecn=off,guest_ufo=on,host_tso4=off,host_tso6=on,host_ecn=on,mrg_rxbuf=off,ctrl_vq=on,host_ufo=on,mrg_rxbuf=on,ctrl_rx=on,ctrl_vlan=on,ctrl_rx_extra=on,ctrl_mac_addr=on -netdev tap,id=hostnet1,vhost=off,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,ifname=fuxc-net1 -device e1000,netdev=hostnet1,id=virtio-net-pci1,mac=00:01:02:03:04:06,bus=pci.0,addr=0xa,multifunction=off -netdev tap,id=hostnet2,vhost=off,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,ifname=fuxc-net-rtl8139 -device rtl8139,netdev=hostnet2,id=virtio-net-pci2,mac=00:01:02:03:04:07,bus=pci.0,addr=0xb,multifunction=off -serial unix:/tmp/monitor2,server,nowait -rtc base=utc -drive file=/mnt/ide-disk,if=none,id=drive-data-disk,format=raw,cache=none,aio=native,werror=stop,rerror=stop,copy-on-read=off,serial=fux-ide,media=disk -device ide-drive,drive=drive-data-disk,id=system-disk,wwn=0x5000c50015ea71ad,logical_block_size=512,physical_block_size=512,min_io_size=32,opt_io_size=64,discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0  -chardev tty,id=serial1,path=/dev/ttyS0 -device isa-serial,chardev=serial1 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait  -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0 -chardev file,id=channel3,path=/mnt/helloworld1.txt -device virtserialport,chardev=channel3,name=com.redhat.rhevm.vdsm1,bus=virtio-serial0.0,id=port1,nr=1 -chardev socket,id=isa-serial-1,path=/tmp/isa-serial-1,server,nowait -device isa-serial,chardev=isa-serial-1 -device ich9-usb-uhci1,id=usb1,addr=0x11 -device usb-tablet,id=input0 -device usb-mouse,id=input1 -global pvpanic.ioport=0x0505 -machine rhel6.5.0,dump-guest-core=off -drive file=/mnt/driver.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-0,id=ide0-1-0,wwn=0x5000c50015ea71ade,logical_block_size=512,physical_block_size=512,min_io_size=32,opt_io_size=64,discard_granularity=512,unit=1,ver=fuxc-ver-cdrom,bus=ide.0,unit=1 -drive file=/dev/sdd,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=0x13 -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=data-disk2,bootindex=0 -device sga -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 -device intel-hda,id=sound0,bus=pci.0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev socket,path=/tmp/foo,server,nowait,id=foo -drive file=/usr/share/virtio-win/virtio-win_amd64.vfd,if=none,id=drive-fdc0-0-0,readonly=on,format=raw -global isa-fdc.driveA=drive-fdc0-0-0  -vnc :10