Bug 1265076

Hi Jinzhao,

Can you give a test and identify whether it is a regression bz?

Best Regards,
Junyi

(In reply to juzhang from comment #3)
> Hi Jinzhao,
> 
> Can you give a test and identify whether it is a regression bz?
> 
> Best Regards,
> Junyi

 ---Verified it with Rhel 7.1 and the issue didn't can be reproduced, following is the details for verifying 
   Guest： win2008 R2 & win8
   Host kernel: 3.10.0-229.el7.x86_64
   qemu-kvm-rhev-2.1.2-23.el7
   
Test steps:
1. boot vm in src host with cli:
      -M pc \
-cpu SandyBridge,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-nodefaults -rtc base=utc -no-hpet \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0 \
-netdev tap,id=hostnet0,vhost=on,queues=4 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=0e:eb:75:b6:4d:ae,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-object iothread,id=iothread0 \
-drive file=/mnt/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1,iothread=iothread0 \
-drive file=/mnt/win8r2/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0,format=raw \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=0 \
-drive file=/usr/share/virtio-win/virtio-win.iso,if=none,media=cdrom,id=drive-ide1,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide1,id=ide1 \

2. boot vm in dest host with cli:
-M pc \
-cpu SandyBridge,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-nodefaults -rtc base=utc -no-hpet \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7.2 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-monitor stdio \
-qmp tcp:0:6661,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/serial0,server,nowait \
-vga cirrus \
-vnc :0  \
-netdev tap,id=hostnet0,vhost=on,fds=20:21:22:23 20<>/dev/tap5 21<>/dev/tap5 22<>/dev/tap5 23<>/dev/tap5 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=0e:eb:75:b6:4d:ae,bus=pci.0,addr=0x3,vectors=10,mq=on \
-chardev socket,id=seabios,path=/tmp/seabios,server,nowait \
-device isa-debugcon,chardev=seabios,iobase=0x402 \
-usb \
-device usb-tablet,id=tablet0 \
-object iothread,id=iothread0 \
-drive file=/mnt/win8r2/win8r2.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,bus=pci.0,id=drive-virtio-disk0,drive=drive-virtio-disk0,bootindex=1,iothread=iothread0 \
-drive file=/mnt/win8r2/en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_with_sp1_x64_dvd_617601.iso,if=none,media=cdrom,id=drive-ide0,format=raw \
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=0 \
-drive file=/usr/share/virtio-win/virtio-win.iso,if=none,media=cdrom,id=drive-ide1,format=raw \
-device ide-drive,bus=ide.0,unit=1,drive=drive-ide1,id=ide1 \
-incoming tcp:0:5800 \

3.migrate -d tcp:destip:5800 when guest enter into the screen of "Where do you want to install windows"

4.Guest can be installed after migration

Reproduced the issue with following combination(win2008 R2 guest) 
1)qemu-kvm-rhev-2.3.0-23.el7 & kernel 3.10.0-316.el7.x86_64
2)qemu-kvm-rhev-2.3.0-15.el7 & kernel 3.10.0-316.el7.x86_64
3)qemu-kvm-rhev-2.3.0-10.el7 & kernel 3.10.0-316.el7.x86_64
4)qemu-kvm-rhev-2.3.0-2.el7 & kernel 3.10.0-316.el7.x86_64
5)qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-316.el7.x86_64
6)qemu-kvm-rhev-2.1.2-23.el7 & kernel3.10.0-315.el7.x86_64
7)qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-310.el7.x86_64

The issue didn't reproduced with following combination (win2008 R2)
qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-229.el7.x86_64

Also, linux guest didn't reproduced the issue with  qemu-kvm-rhev-2.3.0-23.el7 & kernel 3.10.0-316.el7.x86_64 

According to the above test, I guest it's a kernel issue.

Thanks 
Jing Zhao

(In reply to jingzhao from comment #5)
> Reproduced the issue with following combination(win2008 R2 guest) 
> 1)qemu-kvm-rhev-2.3.0-23.el7 & kernel 3.10.0-316.el7.x86_64
> 2)qemu-kvm-rhev-2.3.0-15.el7 & kernel 3.10.0-316.el7.x86_64
> 3)qemu-kvm-rhev-2.3.0-10.el7 & kernel 3.10.0-316.el7.x86_64
> 4)qemu-kvm-rhev-2.3.0-2.el7 & kernel 3.10.0-316.el7.x86_64
> 5)qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-316.el7.x86_64
> 6)qemu-kvm-rhev-2.1.2-23.el7 & kernel3.10.0-315.el7.x86_64
> 7)qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-310.el7.x86_64
> 
> The issue didn't reproduced with following combination (win2008 R2)
> qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-229.el7.x86_64
> 
> Also, linux guest didn't reproduced the issue with 
> qemu-kvm-rhev-2.3.0-23.el7 & kernel 3.10.0-316.el7.x86_64 
> 
> According to the above test, I guest it's a kernel issue.

OK, that's really odd; can you try and narrow down which kernel it is please;
Just try win2008 R2 guest, on qemu-kvm-rhev-2.1.2-23 but try and find
kernels between -229 and -310.

Thanks,

Dave
> 
> Thanks 
> Jing Zhao

(In reply to Dr. David Alan Gilbert from comment #7)
> (In reply to jingzhao from comment #5)
> > Reproduced the issue with following combination(win2008 R2 guest) 
> > 1)qemu-kvm-rhev-2.3.0-23.el7 & kernel 3.10.0-316.el7.x86_64
> > 2)qemu-kvm-rhev-2.3.0-15.el7 & kernel 3.10.0-316.el7.x86_64
> > 3)qemu-kvm-rhev-2.3.0-10.el7 & kernel 3.10.0-316.el7.x86_64
> > 4)qemu-kvm-rhev-2.3.0-2.el7 & kernel 3.10.0-316.el7.x86_64
> > 5)qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-316.el7.x86_64
> > 6)qemu-kvm-rhev-2.1.2-23.el7 & kernel3.10.0-315.el7.x86_64
> > 7)qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-310.el7.x86_64
> > 
> > The issue didn't reproduced with following combination (win2008 R2)
> > qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-229.el7.x86_64
> > 
> > Also, linux guest didn't reproduced the issue with 
> > qemu-kvm-rhev-2.3.0-23.el7 & kernel 3.10.0-316.el7.x86_64 
> > 
> > According to the above test, I guest it's a kernel issue.
> 
> OK, that's really odd; can you try and narrow down which kernel it is please;
> Just try win2008 R2 guest, on qemu-kvm-rhev-2.1.2-23 but try and find
> kernels between -229 and -310.
> 
> Thanks,
> 
> Dave
> > 
> > Thanks 
> > Jing Zhao

Dave: I have reproduced the issue with  qemu-kvm-rhev-2.1.2-23.el7 & kernel 3.10.0-301.el7.x86_64 and the issue can be reproduced, there no other version from 229 -301 in brewweb, so I think we hits the issue from kernel 3.10.0-301.el7.x86_64

Thanks 
Jing Zhao

If you finish the installation using keyboard, does the mouse work after reboot?
("Alt+underscored letter", "Alt+b" is back, "space", "tab" and "enter" will allow you to proceed if there are no other problems.)

Relevant patches from 299/300 diff are mostly related to network ...
Can you reproduce when migrating to localhost?  (No need to setup bonding/sharing; just change qmp, serial and vnc sockets on destination.)
If so, does migration using unix socket (something other than tcp) still fail?

Thanks.

Created attachment 1081257 [details]
cpuinfo for source

So far we have:

- mouse as well as keyboard not responsive after migration (keyboard responds with a delay, but installation doesn't finish) (comment 20)

- migration to localhost doesn't show this problem (using tcp or unix sockets) (comment 17)

- qemu-2.3 with new kernels (>298) exhibits problem (comment 12)
  qemu-2.1 with new kernels does not exhibit problem (comment 18)

Looks like networking-related.

Can you try migration with udp sockets instead of tcp?

Thanks

(In reply to Amit Shah from comment #25)
> Looks like networking-related.
> 
> Can you try migration with udp sockets instead of tcp?

... we don't have support for udp-based migration, so this can't work.

Since comment 18, we also have qemu-2.1/qemu-2.3 line of failure ...

Can you reproduce with a minimalized qemu guest?
(You should be able to get into that screen with just the ISO and its drive.  If you can't reproduce, then bisect the device that causes it.)

And/or when flipping virtio devices? (e1000 <-> virtio-net, ...)

It might also be helpful to know what is going on.  Please upload the output of (not recording at the same time)

  trace-cmd record -e 'kvm:*'

and 

  perf record -g -t `pidof qemu-kvm`

and

  stap - <<< 'probe qemu.kvm.simpletrace.* {}' > stap.out

on the destination after the bug has been hit.

10 seconds is plenty, but give less if it still produces a huge file.  Even 1 second is good if we have tens of megabytes of output.

Thanks.

(A reference set, captured when the bug hasn't been hit would be nice as well :)

Hi Qunfang,

Free to update QE contact.

Hi

Could you confirm if you can reproduce it on 7.4, please?
So far:
- removing usb-tablet device fixes it
- moving to one old kernel fixes it
- but moving to an old qemu on new kernel also fixes it
- we haven't get any info about what happens when we remove
  * virtio-net
- my understanding is that without usb-tablet the issue dissapears, but that the mouse goes really slow, no?

Later, Juan.

Agreed.  Thanks. Closing it.