Bug 698891

Summary: [whql][viostor]Virtio block job of "Common_Scenario_Stress_With IO" for win2k8/win7 stopped going on with MSI=0.
Product: Red Hat Enterprise Linux 6 Reporter: dawu
Component: virtio-winAssignee: Vadim Rozenfeld <vrozenfe>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: bcao, juzhang, knoel, lijin, mdeng, michen, qzhang, rhod, syeghiay, tburke, vrozenfe
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-09 08:57:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
2k8-r2-blk-jobstop_commonscenario-9-withMSI=0
none
job hangs
none
screenshot
none
info-pci
none
seetupapi.dev.log none

Description dawu 2011-04-22 07:58:02 UTC
Created attachment 494096 [details]
2k8-r2-blk-jobstop_commonscenario-9-withMSI=0

Description of problem:
For virtio-win-prewhql-0.1-9,Virtio block job of "Common_Scenario_Stress_With IO" for win2k8/win7 stopped going on with MSI=0,no error happened,please refer to the attachment of "2k8-r2-blk-jobstop_commonscenario-9-withMSI=0.png".

Job can finish and pass with MSI=1.

Version-Release number of selected component (if applicable):

virtio-win-prewhql-0.1-9
kernel-2.6.32-120.el6.x86_64
qemu-kvm-0.12.1.2-2.158.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1.start guest with CLI:
  /usr/libexec/qemu-kvm -m 6G -smp 4 -cpu cpu64-rhel6,+x2apic -usbdevice tablet -drive file=win2k8-r2-blk-9.raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,mac=00:10:1a:16:33:26,bus=pci.0,addr=0x4 -boot c -uuid 3a0b28bc-7eb5-4ad7-8c24-69e9520d67f0 -rtc-td-hack -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/monitor-win2k8-r2-blk-9,server,nowait -mon chardev=111a,mode=readline -name win2k8-r2-blk-9- -vnc :1 -drive file=disk1.raw,if=none,id=drive-virtio1,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio1,id=virtio-blk-pci1 -drive file=disk2.raw,if=none,id=drive-virtio2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio2,id=virtio-blk-pci2 -drive file=disk3.raw,if=none,id=drive-virtio3,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio3,id=virtio-blk-pci3 -drive file=disk4.raw,if=none,id=drive-virtio4,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio4,id=virtio-blk-pci4

2.Run job of "Common_Scenario_Stress_With IO".
  
Actual results:
After do S4 several times, job stopped during running script, no any error information, please refer to the attachment "2k8-r2-blk-jobstop_commonscenario-9-withMSI=0.png". Job can finish and pass with MSI=1.

Expected results:
Job should go on without any stop and error.

Additional info:

Comment 1 Vadim Rozenfeld 2011-07-31 09:43:57 UTC
(In reply to comment #0)
> Created attachment 494096 [details]
> 2k8-r2-blk-jobstop_commonscenario-9-withMSI=0
> 
> Description of problem:
> For virtio-win-prewhql-0.1-9,Virtio block job of "Common_Scenario_Stress_With
> IO" for win2k8/win7 stopped going on with MSI=0,no error happened,please refer
> to the attachment of "2k8-r2-blk-jobstop_commonscenario-9-withMSI=0.png".
> 
> Job can finish and pass with MSI=1.
> 
> Version-Release number of selected component (if applicable):
> 
> virtio-win-prewhql-0.1-9
> kernel-2.6.32-120.el6.x86_64
> qemu-kvm-0.12.1.2-2.158.el6.x86_64
> 
> How reproducible:
> always
> 
> Steps to Reproduce:
> 1.start guest with CLI:
>   /usr/libexec/qemu-kvm -m 6G -smp 4 -cpu cpu64-rhel6,+x2apic -usbdevice 

Hi Dawn,
Could you please check, if the problem is reproducible with 4G and 2G memory options?

Thank you,
Vadim.

Comment 2 Min Deng 2011-08-04 07:00:10 UTC
Hi Vadim,
   I already submit one job for this bug yesterday but the job doesn't finish so far.
/usr/libexec/qemu-kvm -m 4G -smp 2 -cpu cpu64-rhel6,+x2apic -usbdevice tablet -drive file=win2k8-R2-blk.raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,mac=00:13:25:11:45:16,bus=pci.0,addr=0x4 -boot c -uuid bb5174f0-bbcf-4b3d-b153-bc29087ffbfd -rtc-td-hack -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/monitor-win2k8-R2-blk,server,nowait -mon chardev=111a,mode=readline -name win2k8-R2-blk -drive file=disk1.raw,if=none,id=drive-virtio1,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio1,id=virtio-blk-pci1 -drive file=disk2.raw,if=none,id=drive-virtio2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio2,id=virtio-blk-pci2 -drive file=disk3.raw,if=none,id=drive-virtio3,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio3,id=virtio-blk-pci3 -drive file=disk4.raw,if=none,id=drive-virtio4,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio4,id=virtio-blk-pci4 -spice port=5931,disable-ticketing -vga qxl

Thanks
Min Deng

Comment 3 Min Deng 2011-08-05 02:05:13 UTC
Hi Vadim,

   It seems that the job hangs and please see the snapshot.and I have to re-submit new one.And I just change the Memory to 4G this time.

Thanks 
Min Deng

Comment 4 Min Deng 2011-08-05 02:09:28 UTC
Created attachment 516809 [details]
job hangs

Comment 5 Min Deng 2011-08-08 02:09:53 UTC
Actually,the issue I hit (Memory equal 4G) is the same to the bug.please look into the two screen shots carefully.

Thanks,
Min

Comment 6 Vadim Rozenfeld 2011-09-12 08:45:23 UTC
Hi Min,
Could you please check this issue again on a freshly installed system and the 
latest drivers? Please make sure that we have correct time-zone and rtc=localtime
in the command line.

Best regards,
Vadim.

Comment 7 Min Deng 2011-09-14 09:17:45 UTC
Hi Vadim,

   Already submit a job of windows 2k8-R2 for the bug.

Thanks
Min

Comment 8 Min Deng 2011-09-15 01:57:09 UTC
Command line is as the followings
/usr/libexec/qemu-kvm -m 6G -smp 4 -cpu cpu64-rhel6,+x2apic -usb -device usb-tablet -drive file=win2k8-R2-block.raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop,format=raw -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,mac=00:11:cF:26:23:51,bus=pci.0,addr=0x4 -boot c -uuid 2ce82702-6be5-40a7-a96e-c29c5f729ab7 -rtc-td-hack -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/monitor-2k8-32-blk-sluo,server,nowait -mon chardev=111a,mode=readline -name 2k8-R2-block -vnc :1 -drive file=disk1.raw,if=none,id=drive-virtio1,cache=none,werror=stop,rerror=stop,format=raw -device virtio-blk-pci,drive=drive-virtio1,id=virtio-blk-pci1 -drive file=disk2.raw,if=none,id=drive-virtio2,cache=none,werror=stop,rerror=stop,format=raw -device virtio-blk-pci,drive=drive-virtio2,id=virtio-blk-pci2 -drive file=disk3.raw,if=none,id=drive-virtio3,cache=none,werror=stop,rerror=stop,format=raw -device virtio-blk-pci,drive=drive-virtio3,id=virtio-blk-pci3 -drive file=disk4.raw,if=none,id=drive-virtio4,cache=none,werror=stop,rerror=stop,format=raw -device virtio-blk-pci,drive=drive-virtio4,id=virtio-blk-pci4 -rtc base=localtime

It still cannot stop and see the screenshot 2k8-R2

Comment 9 Min Deng 2011-09-15 01:59:38 UTC
Created attachment 523291 [details]
screenshot

 I will change anther disk to run the job again

Comment 10 Min Deng 2011-09-15 02:57:30 UTC
Hi Vadim,

   Other hits the same issue on windows 2008 32 bits guest,had better re-check the issue.thanks.

Best Regards
Min Deng

Comment 11 Vadim Rozenfeld 2011-09-15 05:34:11 UTC
(In reply to comment #10)
> Hi Vadim,
> 
>    Other hits the same issue on windows 2008 32 bits guest,had better re-check
> the issue.thanks.
> 
> Best Regards
> Min Deng

Hi Min,

What happens with VM itself, when the test is hung? Is it continue to be responsive or hung also?

I would also ask you for the following information:
"pci info", cpk file, setupapi.dev.log files.

Thank you,
Vadim.

Comment 13 Mike Cao 2011-10-20 02:43:40 UTC
Hi, Vadim

I want to ask some questions about this bug .
What does MSI used for ? 
What's the different of MSI=1 and MSI=0,Which one is better for QE to run WHQL? 
Since We passed blk whql with virtio-win-prewhql-0.1.16,what's the default value of MSI in that build ? 

TIA,
Mike

Comment 14 Vadim Rozenfeld 2011-10-20 04:56:06 UTC
(In reply to comment #13)
> Hi, Vadim
> 
> I want to ask some questions about this bug .
> What does MSI used for ? 
> What's the different of MSI=1 and MSI=0,Which one is better for QE to run WHQL? 
> Since We passed blk whql with virtio-win-prewhql-0.1.16,what's the default
> value of MSI in that build ? 
> 
> TIA,
> Mike

Hi Mike,

MSI is just another way of generating an interrupt in the system.
If MSI=1, Operating System is trying to allocate and use MSI instead 
of IRQ. Technically, MSI and IRQ are almost equal, with only one difference 
that MSI should be a little faster and use less CPU consumption.

Best,
Vadim.

Comment 15 Mike Cao 2011-10-20 05:08:03 UTC
(In reply to comment #14)

> 
> Hi Mike,
> 
> MSI is just another way of generating an interrupt in the system.
> If MSI=1, Operating System is trying to allocate and use MSI instead 
> of IRQ. Technically, MSI and IRQ are almost equal, with only one difference 
> that MSI should be a little faster and use less CPU consumption.
> 
> Best,
> Vadim.

Hi, Vadim

Do you mean now we support both (MSI=0 and 1) ?
Then what about this bug ,this issue is still existed in virtio-win-prewhql-0.1.16 if using MSI=0 .

Best Regards,
Mike

Comment 16 Vadim Rozenfeld 2011-10-20 08:48:30 UTC
(In reply to comment #15)
> (In reply to comment #14)
> 
> > 
> > Hi Mike,
> > 
> > MSI is just another way of generating an interrupt in the system.
> > If MSI=1, Operating System is trying to allocate and use MSI instead 
> > of IRQ. Technically, MSI and IRQ are almost equal, with only one difference 
> > that MSI should be a little faster and use less CPU consumption.
> > 
> > Best,
> > Vadim.
> 
> Hi, Vadim
> 
> Do you mean now we support both (MSI=0 and 1) ?
> Then what about this bug ,this issue is still existed in
> virtio-win-prewhql-0.1.16 if using MSI=0 .
> 
> Best Regards,
> Mike

Yes, we must be able to support both cases: MSI (MSI=1) and IRQ (MSI=0).
There is something weird about this bug. Viostor driver supports MSI starting
from 6.0, where it was turned off be default. And we didn't see any problem,
passing WHQL at that time. In 6.1 we switched MSI on without making any
interrupt handling changes in the driver, but suddenly IRQ (MSI=0) mode starts
making problems. Frankly, I cannot find any logical explanation why MSI 
works fine in 6.1/6.2, while IRQ doesn't. 
  
Best regards,
Vadim.

Comment 17 Min Deng 2011-10-20 09:32:49 UTC
Hi All,

 I re-test the bug with virtio-win-prewhql-0.1.16 and still can reproduce the issue.
 Two ways for change MSI to IRQ. 
 a.change visotor.inf (1->0) before installing driver directly.
 b.change Registry of guest after installing 
CLI
/usr/libexec/qemu-kvm -m 6G -smp 4 -cpu cpu64-rhel6,+x2apic,family=0xf -usb -device usb-tablet -drive file=win2k8-R2-block.raw,format=raw,if=none,id=drive-virtio0,boot=on,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio0,id=virtio-blk-pci0,bootindex=1 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,mac=00:11:25:c1:c1:46,bus=pci.0,addr=0x4 -uuid 70a500b3-accc-4e3c-a356-ac2995d827d6 -rtc base=localtime -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/win2k8-R2-blk,server,nowait -mon chardev=111a,mode=readline -name win2k8-R2-blk -drive file=disk1.raw,format=raw,if=none,id=drive-virtio1,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio1,id=virtio-blk-pci1 -drive file=disk2.raw,format=raw,if=none,id=drive-virtio2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio2,id=virtio-blk-pci2 -drive file=disk3.raw,format=raw,if=none,id=drive-virtio3,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio3,id=virtio-blk-pci3 -drive file=disk4.raw,format=raw,if=none,id=drive-virtio4,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio4,id=virtio-blk-pci4 -spice port=5931,disable-ticketing -vga qxl

Thanks,
Min Deng

Comment 18 Vadim Rozenfeld 2011-10-20 09:50:08 UTC
Hi Min,
Could you please post the following information:

"pci info" output just after test was stopped , cpk file, and 
setupapi.dev.log file?

Thank you,
Vadim.

Comment 19 Min Deng 2011-10-21 02:49:32 UTC
(In reply to comment #18)
> Hi Min,
> Could you please post the following information:
> 
> "pci info" output just after test was stopped , cpk file, and 
> setupapi.dev.log file?
> 
> Thank you,
> Vadim.

Hi Vadim,

   I stopped the job manually and also found the guest hardly shut down itself.
I had to reset it from monitor.
   The "pci info",cpk file and setupapi.dev.log,please see attachments.Any issues please let me know.thanks

Best Regards,
Min

Comment 20 Min Deng 2011-10-21 02:50:07 UTC
Created attachment 529410 [details]
info-pci

Comment 21 Min Deng 2011-10-21 02:50:56 UTC
Created attachment 529411 [details]
CPK

Comment 22 Min Deng 2011-10-21 02:51:30 UTC
Created attachment 529412 [details]
seetupapi.dev.log

Comment 23 Mike Cao 2011-10-21 05:19:45 UTC
Based on comment #17 ,re-assign this issue .
Since we passed RHEL6.2 whql process ,I suggests to move this issue to 6.3

Comment 27 Mike Cao 2013-01-04 05:12:02 UTC
Vadim

Since We usually use MSi=1 to pass whql certification ,Do you still want to fix this bug ?

Thanks,
Mike

Comment 28 Vadim Rozenfeld 2013-01-04 11:08:53 UTC
Hi Mike.
This bug definitely has the lowest priority.
I think the problem happens because by default we declare that we support MSI
and then turn it off, which is slightly unusual situation. W2K3 always has MSI disabled by default and still can pass WHQL. 
Mike, could you please try checking the following scenario, when we install Win7
and check viostor driver on a system with cpu family less than 0xf? In this situation MSI support will be disabled by OS, even though we still have MSI=1 in the Registry.   

Thank you,
Vadim.

Comment 29 Mike Cao 2013-01-08 02:48:05 UTC
(In reply to comment #28)
> Hi Mike.
> This bug definitely has the lowest priority.
> I think the problem happens because by default we declare that we support MSI
> and then turn it off, which is slightly unusual situation. W2K3 always has
> MSI disabled by default and still can pass WHQL. 
> Mike, could you please try checking the following scenario, when we install
> Win7
> and check viostor driver on a system with cpu family less than 0xf? In this
> situation MSI support will be disabled by OS, even though we still have
> MSI=1 in the Registry.   

I will try -cpu cpu64-rhel6,+x2apic,family=0x1 ,Is it ok to disable MSI ?

Best Regards,
Mike
> 
> Thank you,
> Vadim.

Comment 30 Vadim Rozenfeld 2013-01-08 08:16:26 UTC
(In reply to comment #29)
> (In reply to comment #28)
> > Hi Mike.
> > This bug definitely has the lowest priority.
> > I think the problem happens because by default we declare that we support MSI
> > and then turn it off, which is slightly unusual situation. W2K3 always has
> > MSI disabled by default and still can pass WHQL. 
> > Mike, could you please try checking the following scenario, when we install
> > Win7
> > and check viostor driver on a system with cpu family less than 0xf? In this
> > situation MSI support will be disabled by OS, even though we still have
> > MSI=1 in the Registry.   
> 
> I will try -cpu cpu64-rhel6,+x2apic,family=0x1 ,Is it ok to disable MSI ?
Windows disables MSI support for platforms if cpu family is lower than 0xf.
But I'm just not sure about such low number. Try it with family=0xe. It will definitely do the trick.
Best regards,
Vadim.   
> 
> Best Regards,
> Mike
> > 
> > Thank you,
> > Vadim.

Comment 31 Mike Cao 2013-01-08 10:02:25 UTC
(In reply to comment #30)
> (In reply to comment #29)
> > (In reply to comment #28)
> > > Hi Mike.
> > > This bug definitely has the lowest priority.
> > > I think the problem happens because by default we declare that we support MSI
> > > and then turn it off, which is slightly unusual situation. W2K3 always has
> > > MSI disabled by default and still can pass WHQL. 
> > > Mike, could you please try checking the following scenario, when we install
> > > Win7
> > > and check viostor driver on a system with cpu family less than 0xf? In this
> > > situation MSI support will be disabled by OS, even though we still have
> > > MSI=1 in the Registry.   
> > 
> > I will try -cpu cpu64-rhel6,+x2apic,family=0x1 ,Is it ok to disable MSI ?
> Windows disables MSI support for platforms if cpu family is lower than 0xf.
> But I'm just not sure about such low number. Try it with family=0xe. It will
> definitely do the trick.

job can pass with family=0x1 ,Let me try family=0xe now

Comment 32 Mike Cao 2013-01-09 03:04:18 UTC
(In reply to comment #31)
> (In reply to comment #30)
> > (In reply to comment #29)
> > > (In reply to comment #28)
> > > > Hi Mike.
> > > > This bug definitely has the lowest priority.
> > > > I think the problem happens because by default we declare that we support MSI
> > > > and then turn it off, which is slightly unusual situation. W2K3 always has
> > > > MSI disabled by default and still can pass WHQL. 
> > > > Mike, could you please try checking the following scenario, when we install
> > > > Win7
> > > > and check viostor driver on a system with cpu family less than 0xf? In this
> > > > situation MSI support will be disabled by OS, even though we still have
> > > > MSI=1 in the Registry.   
> > > 
> > > I will try -cpu cpu64-rhel6,+x2apic,family=0x1 ,Is it ok to disable MSI ?
> > Windows disables MSI support for platforms if cpu family is lower than 0xf.
> > But I'm just not sure about such low number. Try it with family=0xe. It will
> > definitely do the trick.
> 
> job can pass with family=0x1 ,Let me try family=0xe now

Tried it with win7-64 bit guest and win2k8R2 guests ,both can not reproduce this issue .Close it ?

Best Regards,
Mike

Comment 33 Vadim Rozenfeld 2013-01-09 08:36:06 UTC
(In reply to comment #32)
> (In reply to comment #31)
> > (In reply to comment #30)
> > > (In reply to comment #29)
> > > > (In reply to comment #28)
> > > > > Hi Mike.
> > > > > This bug definitely has the lowest priority.
> > > > > I think the problem happens because by default we declare that we support MSI
> > > > > and then turn it off, which is slightly unusual situation. W2K3 always has
> > > > > MSI disabled by default and still can pass WHQL. 
> > > > > Mike, could you please try checking the following scenario, when we install
> > > > > Win7
> > > > > and check viostor driver on a system with cpu family less than 0xf? In this
> > > > > situation MSI support will be disabled by OS, even though we still have
> > > > > MSI=1 in the Registry.   
> > > > 
> > > > I will try -cpu cpu64-rhel6,+x2apic,family=0x1 ,Is it ok to disable MSI ?
> > > Windows disables MSI support for platforms if cpu family is lower than 0xf.
> > > But I'm just not sure about such low number. Try it with family=0xe. It will
> > > definitely do the trick.
> > 
> > job can pass with family=0x1 ,Let me try family=0xe now
> 
> Tried it with win7-64 bit guest and win2k8R2 guests ,both can not reproduce
> this issue .Close it ?
Yes.
Thank you,
Vadim.
> 
> Best Regards,
> Mike