Bug 811841

Created attachment 576949 [details]
win7-32-S4AfterHotUnplugOrPlug-Fail2

Hi, Vadim 

more info for this bug ,(might be related to MSFT) 


I also tried following steps:

1.start guests w/ 2 virito disks 
<CLI>
-drive file=/home/win2k8R2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none 
-device virtio-blk-pci,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/home/hotadd.img,format=qcow2,if=none,werror=stop,rerror=stop,cache=none,id=driver_hotadd 
-device virtio-blk-pci,id=hotadd,drive=driver_hotadd,addr=0xa

2.hot-unplug virtio disk named hotadd 
3.s4 guest
4
*if* start guest w/o the hot-unplug disk ,will hit the issue described in comment #0 ,guest also complains "your system's firmware did not preserve the system memory map across the hibernate transition " 

*if* start guest commandLine exactly same as steps 1 ,guest could resume successfully.


Additional info :
Tried w/ win2k8R2

(In reply to comment #4)
> Hi, Vadim 
> 
> more info for this bug ,(might be related to MSFT) 
> 
> 
> I also tried following steps:
> 
> 1.start guests w/ 2 virito disks 
> <CLI>
> -drive file=/home/win2k8R2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none 
> -device virtio-blk-pci,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -drive
> file=/home/hotadd.img,format=qcow2,if=none,werror=stop,rerror=stop,cache=none,id=driver_hotadd 
> -device virtio-blk-pci,id=hotadd,drive=driver_hotadd,addr=0xa
> 
> 2.hot-unplug virtio disk named hotadd 
> 3.s4 guest
> 4
> *if* start guest w/o the hot-unplug disk ,will hit the issue described in
> comment #0 ,guest also complains "your system's firmware did not preserve the
> system memory map across the hibernate transition " 
> 
> *if* start guest commandLine exactly same as steps 1 ,guest could resume
> successfully.
> 
> 
> Additional info :
> Tried w/ win2k8R2

This interesting thing also happened on win7-32.

The issue seems to be unrelated to the Unplug itself.
The issue is that the hardware should not change during S4 (controllers and devices should not be added/deleted).
If Windows detects that the hardware changed, it warns the user exactly as seen in the attached screen dump.
Since the management software should take care of this, I believe that both virt-manager and RHEV-M, will survive this scenario, and resume properly after S4 without the unplugged device. Worth verifying.
Anyhow, this is not a QEMU/virtio-win bug, so I am closing it.

(In reply to comment #4)
> Hi, Vadim 
> 
> more info for this bug ,(might be related to MSFT) 
> 
> 
> I also tried following steps:
> 
> 1.start guests w/ 2 virito disks 
> <CLI>
> -drive file=/home/win2k8R2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none 
> -device virtio-blk-pci,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -drive
> file=/home/hotadd.img,format=qcow2,if=none,werror=stop,rerror=stop,cache=none,id=driver_hotadd 
> -device virtio-blk-pci,id=hotadd,drive=driver_hotadd,addr=0xa
> 
> 2.hot-unplug virtio disk named hotadd 
> 3.s4 guest
> 4
> *if* start guest w/o the hot-unplug disk ,will hit the issue described in
> comment #0 ,guest also complains "your system's firmware did not preserve the
> system memory map across the hibernate transition " 

Hi Mike, I missed this point. I will take a more precise look into this issue,
but in any case, it is not a viostor but a QEMU problem.
Thank you,
Vadim.

> 
> *if* start guest commandLine exactly same as steps 1 ,guest could resume
> successfully.
> 
> 
> Additional info :
> Tried w/ win2k8R2

(In reply to comment #7)
> (In reply to comment #4)
> > Hi, Vadim 
> > 
> > more info for this bug ,(might be related to MSFT) 
> > 
> > 
> > I also tried following steps:
> > 
> > 1.start guests w/ 2 virito disks 
> > <CLI>
> > -drive file=/home/win2k8R2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none 
> > -device virtio-blk-pci,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> > -drive
> > file=/home/hotadd.img,format=qcow2,if=none,werror=stop,rerror=stop,cache=none,id=driver_hotadd 
> > -device virtio-blk-pci,id=hotadd,drive=driver_hotadd,addr=0xa
> > 
> > 2.hot-unplug virtio disk named hotadd 
> > 3.s4 guest
> > 4
> > *if* start guest w/o the hot-unplug disk ,will hit the issue described in
> > comment #0 ,guest also complains "your system's firmware did not preserve the
> > system memory map across the hibernate transition " 
> 
> Hi Mike, I missed this point. I will take a more precise look into this issue,
> but in any case, it is not a viostor but a QEMU problem.
> Thank you,
> Vadim.
> 

Hi ,Vadim

I talked with Amit about this bug  .he think this is a windows guest bug instead of qemu one  .I did not re-open this bug now because I want to check whether netkvm and balloon has such ab-normal behaviour 

more info about this bug :

If I reboot guest after hot-unplug device (step 2 in comment  #0) and before hibernate guest (step 3 in comment #0)  ,then I will not hit this bug 

Mike

dawu also tried netkvm and balloon driver ,did not hit issue

Based on above and comment #8 ,this should be a block driver bug .re-open it

And since s4 will not support during rhel6.3 ,I strongly suggest we can move this bug to rhel6.4.0 


Mike

Mike,
Thanks for reopening. Vadim and I mistakenly thought that the unplugged disk were present at "resume".
Anyhow, Since it is not a regression, at this stage, we better move it to 6.4. Vadim will continue to look at it, so there is a chance that it will make it to 6.3.

I try balloon again
after guest hibernate ,guest could resume successfully both w/ -device virtio-balloon-pci,id=balloon.0 and w/o -device virtio-balloon-pci

Mike, Dawn
Can we try the following scenario:
- start qemu with "-no-shutdown" flag
- do hot plug or unplug
- hibernate the VM
- resume VM ('system reset', 'cont' from the qemu monitor)

Thank you,
Vadim.

(In reply to comment #18)
> Mike, Dawn
> Can we try the following scenario:
> - start qemu with "-no-shutdown" flag
> - do hot plug or unplug
> - hibernate the VM
> - resume VM ('system reset', 'cont' from the qemu monitor)
> 
> Thank you,
> Vadim.

Hi, Vadim

Tried on windows server 2008R2
Steps as your desbribed 
1.start VM w/ --no-shutdown
2.hot-unplug the virtio data disk 
3.s4 guest 
4.system_reset
5.cont 

Actual Results:
hit the exactly same issue

(In reply to comment #19)
> (In reply to comment #18)
> > Mike, Dawn
> > Can we try the following scenario:
> > - start qemu with "-no-shutdown" flag
> > - do hot plug or unplug
> > - hibernate the VM
> > - resume VM ('system reset', 'cont' from the qemu monitor)
> > 
> > Thank you,
> > Vadim.
> 
> Hi, Vadim
> 
> Tried on windows server 2008R2
> Steps as your desbribed 
> 1.start VM w/ --no-shutdown
> 2.hot-unplug the virtio data disk 
> 3.s4 guest 
> 4.system_reset
> 5.cont 
> 
> Actual Results:
> hit the exactly same issue

Hi Mike,
Could you please post pci resources ('info pci') before and hibernation and after resume?
Thank you,
Vadim.

Before hibernation
(qemu) info pci 
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1b36:0100
      IRQ 10.
      BAR0: 32 bit memory at 0xf0000000 [0xf3ffffff].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x03fffffe].
      BAR2: 32 bit memory at 0xf4000000 [0xf4003fff].
      BAR3: I/O at 0xc020 [0xc03f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
      id "ide0-0-0"
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc080 [0xc0bf].
      BAR1: 32 bit memory at 0xf4021000 [0xf4021fff].
      id "ide0-1-0"
  Bus  0, device   5, function 0:
    Class 0403: PCI device 8086:2668
      IRQ 10.
      BAR0: 32 bit memory at 0xf4024000 [0xf4027fff].
      id "sound0"
  Bus  0, device   6, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 10.
      BAR0: I/O at 0xc0c0 [0xc0df].
      id "balloon"
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 10.
      BAR0: 32 bit memory at 0xf4040000 [0xf405ffff].
      BAR1: I/O at 0xc100 [0xc13f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "net0"

After Device_del
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1b36:0100
      IRQ 5.
      BAR0: 32 bit memory at 0xf0000000 [0xf3ffffff].
      BAR1: 32 bit memory at 0xf8000000 [0xfbffffff].
      BAR2: 32 bit memory at 0xf4000000 [0xf4003fff].
      BAR3: I/O at 0xc020 [0xc03f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
      id "ide0-0-0"
  Bus  0, device   5, function 0:
    Class 0403: PCI device 8086:2668
      IRQ 11.
      BAR0: 32 bit memory at 0xf4024000 [0xf4027fff].
      id "sound0"
  Bus  0, device   6, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 5.
      BAR0: I/O at 0xc0c0 [0xc0df].
      id "balloon"
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 11.
      BAR0: 32 bit memory at 0xf4040000 [0xf405ffff].
      BAR1: I/O at 0xc100 [0xc13f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "net0"

hibernate the VM
info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1b36:0100
      IRQ 5.
      BAR0: 32 bit memory at 0xf0000000 [0xf3ffffff].
      BAR1: 32 bit memory at 0xf8000000 [0xfbffffff].
      BAR2: 32 bit memory at 0xf4000000 [0xf4003fff].
      BAR3: I/O at 0xc020 [0xc03f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
      id "ide0-0-0"
  Bus  0, device   5, function 0:
    Class 0403: PCI device 8086:2668
      IRQ 11.
      BAR0: 32 bit memory at 0xffffffffffffffff [0x00003ffe].
      id "sound0"
  Bus  0, device   6, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 5.
      BAR0: I/O at 0xffffffffffffffff [0x001e].
      id "balloon"
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 11.
      BAR0: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      BAR1: I/O at 0xffffffffffffffff [0x003e].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "net0"

(qemu)system_reset
(qemu)cont
(qemu)info pci
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc000 [0xc00f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1b36:0100
      IRQ 10.
      BAR0: 32 bit memory at 0xf0000000 [0xf3ffffff].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x03fffffe].
      BAR2: 32 bit memory at 0xf4000000 [0xf4003fff].
      BAR3: I/O at 0xc020 [0xc03f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
      id "ide0-0-0"
  Bus  0, device   5, function 0:
    Class 0403: PCI device 8086:2668
      IRQ 10.
      BAR0: 32 bit memory at 0xf4024000 [0xf4027fff].
      id "sound0"
  Bus  0, device   6, function 0:
    RAM controller: PCI device 1af4:1002
      IRQ 10.
      BAR0: I/O at 0xc080 [0xc09f].
      id "balloon"
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 10.
      BAR0: 32 bit memory at 0xf4040000 [0xf405ffff].
      BAR1: I/O at 0xc0c0 [0xc0ff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0001fffe].
      id "net0"

Thanks, seems like system_reset returns the original configuration. 
That is why this trick doesn't work. I'll try to find a better way
how to resume VM with absolutely the same HW configuration as it was on 
hibernate.

Cheers,
Vadim.

(In reply to comment #25)
> Thanks, seems like system_reset returns the original configuration. 
> That is why this trick doesn't work. I'll try to find a better way
> how to resume VM with absolutely the same HW configuration as it was on 
> hibernate.
> 
> Cheers,
> Vadim.

Sorry ,What's the original configuration mean here ?the pci info after hot-unplug ?

(In reply to comment #26)
> (In reply to comment #25)
> > Thanks, seems like system_reset returns the original configuration. 
> > That is why this trick doesn't work. I'll try to find a better way
> > how to resume VM with absolutely the same HW configuration as it was on 
> > hibernate.
> > 
> > Cheers,
> > Vadim.
> 
> Sorry ,What's the original configuration mean here ?the pci info after
> hot-unplug ?

if I read it correctly:
we have 
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
      id "ide0-0-0"
before hibernate, and 
  Bus  0, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc07f].
      BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
      id "ide0-0-0"
after resume. For OS it means that HW configuration was changed between these two events,

(In reply to comment #27)
> (In reply to comment #26)
> > (In reply to comment #25)
> > > Thanks, seems like system_reset returns the original configuration. 
> > > That is why this trick doesn't work. I'll try to find a better way
> > > how to resume VM with absolutely the same HW configuration as it was on 
> > > hibernate.
> > > 
> > > Cheers,
> > > Vadim.
> > 
> > Sorry ,What's the original configuration mean here ?the pci info after
> > hot-unplug ?
> 
> if I read it correctly:
> we have 
>   Bus  0, device   3, function 0:
>     SCSI controller: PCI device 1af4:1001
>       IRQ 0.
>       BAR0: I/O at 0xc040 [0xc07f].
>       BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
>       id "ide0-0-0"
> before hibernate, and 
>   Bus  0, device   3, function 0:
>     SCSI controller: PCI device 1af4:1001
>       IRQ 11.
>       BAR0: I/O at 0xc040 [0xc07f].
>       BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
>       id "ide0-0-0"
> after resume. For OS it means that HW configuration was changed between
> these two events,

I did not find 00:03.0 changed 
What I did is remove 00:04.0

(In reply to comment #28)
> (In reply to comment #27)
> > (In reply to comment #26)
> > > (In reply to comment #25)
> > > > Thanks, seems like system_reset returns the original configuration. 
> > > > That is why this trick doesn't work. I'll try to find a better way
> > > > how to resume VM with absolutely the same HW configuration as it was on 
> > > > hibernate.
> > > > 
> > > > Cheers,
> > > > Vadim.
> > > 
> > > Sorry ,What's the original configuration mean here ?the pci info after
> > > hot-unplug ?
> > 
> > if I read it correctly:
> > we have 
> >   Bus  0, device   3, function 0:
> >     SCSI controller: PCI device 1af4:1001
> >       IRQ 0.
> >       BAR0: I/O at 0xc040 [0xc07f].
> >       BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
> >       id "ide0-0-0"
> > before hibernate, and 
> >   Bus  0, device   3, function 0:
> >     SCSI controller: PCI device 1af4:1001
> >       IRQ 11.
> >       BAR0: I/O at 0xc040 [0xc07f].
> >       BAR1: 32 bit memory at 0xf4020000 [0xf4020fff].
> >       id "ide0-0-0"
> > after resume. For OS it means that HW configuration was changed between
> > these two events,
> 
> I did not find 00:03.0 changed 
> What I did is remove 00:04.0

Got you .you mean IRQ Changed

Hi Mike,
Can yo please recheck this issue with the latest drivers?

Thanks,
Vadim.

QE, please recheck. Thanks.

  QE still could reproduce the bug on rhel6.6 host but I could not reproduce it on rhel7 host.
On rhel6.6 host *reproduce*
virtio-win-prewhql-0.1-89
kernel-2.6.32-492.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.430.el6.x86_64
seabios-0.6.1.2-28.el6.x86_64 (latest so far)

On rhel7 host  *Not reproduce*
virtio-win-prewhql-0.1-89
kernel-3.10.0-143.el7.x86_64
qemu-kvm-rhev-2.1.0-1.el7.x86_64.rpm
seabios-1.7.5-4.el7.x86_64.rpm
seabios-bin-1.7.5-4.el7.noarch.rpm
seavgabios-bin-1.7.5-4.el7.noarch.rpm

Steps,
1.boot up guest /usr/libexec/qemu-kvm -cpu SandyBridge,+x2apic -smp 4 -m 2G -device virtio-balloon-pci,id=balloon0 -k en-us -usb -device usb-tablet,id=tablet0 -drive file=win2k8-32.qcow2,format=qcow2,cache=none,if=none,id=scsi0,media=disk -device virtio-blk-pci,drive=scsi0,id=ide-disk0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=net0,mac=00:12:7a:00:11:12 -rtc base=localtime,clock=host,driftfix=slew -name win2k8 -spice port=5931,disable-ticketing -vga qxl -uuid a4ef4247-36ba-4f37-bf98-e82de9382532 -monitor stdio -boot cd  -drive file=disk1.qcow2,format=qcow2,cache=none,if=none,id=scsi1,media=disk  -device virtio-blk-pci,drive=scsi1,id=ide-disk1  -drive file=disk2.qcow2,format=qcow2,cache=none,if=none,id=scsi2,media=disk -device virtio-blk-pci,drive=scsi2,id=ide-disk2 -drive file=disk3.raw,format=raw,cache=none,if=none,id=scsi3,media=disk -device virtio-blk-pci,drive=scsi3,id=ide-disk3 -global PIIX4_PM.disable_s3=0

2.(qemu) __com.redhat_drive_add  file=test3.raw,format=raw,id=blkdisk3
  (qemu) device_add virtio-blk-pci,drive=blkdisk3,id=blkdisk3
3.Do S4

4.resume guest with CLI of step1 + "-drive file=test3.raw,format=raw,cache=none,if=none,id=blkdisk3,media=disk -device virtio-blk-pci,drive=blkdisk3,id=blkdisk3"


Actual results,hit the bug (only on rhel6.6 host)
Expected results,guest could resume successfully

So could you please double check ? Thanks!

(In reply to dengmin from comment #36)
>   QE still could reproduce the bug on rhel6.6 host but I could not reproduce
> it on rhel7 host.

Since S4 is not officially supported in RHEL6, and it seems to be solved in RHEL7, I am closing this bug.
Thanks.