Bug 1157987 - Windows hangs at startup if virtio-scsi device is configured with vectors=1, 2, and 3
Summary: Windows hangs at startup if virtio-scsi device is configured with vectors=1, ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virtio-win
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 7.3
Assignee: Ladi Prosek
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-28 08:18 UTC by ShupingCui
Modified: 2016-11-04 08:43 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
NO_DOCS
Clone Of:
Environment:
Last Closed: 2016-11-04 08:43:44 UTC
Target Upstream Version:


Attachments (Terms of Use)
screenshot (11.69 KB, image/jpeg)
2014-10-28 08:19 UTC, ShupingCui
no flags Details
win7-32 screendump (18.39 KB, image/jpeg)
2015-01-06 01:59 UTC, ShupingCui
no flags Details
win7-64 installation screenshot (190.77 KB, image/png)
2015-12-30 03:06 UTC, Peixiu Hou
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2609 normal SHIPPED_LIVE virtio-win bug fix and enhancement update 2016-11-03 15:27:12 UTC

Description ShupingCui 2014-10-28 08:18:39 UTC
Description of problem:
Win7-64 guest can not install with '-M rhel6.4.0' and virtio-scsi and 3 cdroms, always on 'Setup is starting'

Version-Release number of selected component (if applicable):
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1. install Win7-64 guest with '-M rhel6.4.0' and virtio-scsi and 3 cdroms
/usr/bin/qemu-kvm \
    -name 'virt-tests-vm1' \
    -M rhel6.4.0  \
    -nodefaults  \
    -vga qxl  \
    -chardev socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20141028-125749-BHSUY4UR,server,nowait \
    -mon chardev=hmp_id_hmp1,mode=readline  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20141028-125749-BHSUY4UR,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20141028-125749-BHSUY4UR,path=/tmp/seabios-20141028-125749-BHSUY4UR,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20141028-125749-BHSUY4UR,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 \
    -drive id=drive_image1,if=none,aio=native,file=/mnt/nfs_images/win7-64-sp1-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:e5:55:b4:f9:e8,id=idPIcjJN,vectors=4,netdev=idG0P6Vq,bus=pci.0,addr=05  \
    -netdev tap,id=idG0P6Vq,fd=24  \
    -m 4096  \
    -smp 14,maxcpus=14,cores=1,threads=2,sockets=7  \
    -cpu 'Opteron_G3' \
    -drive id=drive_cd1,if=none,aio=native,media=cdrom,file=/home/kvm_autotest_root/iso/ISO/Windows7/en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso \
    -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -drive id=drive_winutils,if=none,aio=native,media=cdrom,file=/usr/local/autotest/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-drive,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 \
    -drive id=drive_unattended,if=none,aio=native,media=cdrom,file=/usr/local/autotest/tests/virt/shared/data/images/win7-64-sp1/autounattend.iso \
    -device ide-drive,id=unattended,drive=drive_unattended,bus=ide.1,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,disable-ticketing  \
    -rtc base=localtime,clock=host,driftfix=none  \
    -boot order=cdn,once=d,menu=off \
    -enable-kvm

2.
3.

Actual results:
always on 'Setup is starting', cannot install successfully

Expected results:
can install successfully

Additional info:

1. '-M rhel6.5.0' and virtio-scsi and 3 cdroms -> PASSED
2. '-M rhel6.4.0' and virtio-blk and 3 cdroms -> PASSED
3. '-M rhel6.4.0' and virtio-scsi and 3 cdroms -> FAILED
4. '-M rhel6.4.0' and ide and 3 cdroms -> PASSED

this bug can reproduce on qemu-kvm-rhev-0.12.1.2-2.415.el6

Comment 1 ShupingCui 2014-10-28 08:19:10 UTC
Created attachment 951307 [details]
screenshot

Comment 3 ShupingCui 2015-01-06 01:59:29 UTC
Created attachment 976671 [details]
win7-32 screendump

win7-32 met it too, and stuck here.

cmd line:
/usr/bin/qemu-kvm \
    -name 'virt-tests-vm1' \
    -M rhel6.4.0  \
    -nodefaults  \
    -vga qxl \
    -device AC97,bus=pci.0,addr=03  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20150106-092613-B9fc4yTR,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20150106-092613-B9fc4yTR,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20150106-092613-B9fc4yTR,path=/tmp/seabios-20150106-092613-B9fc4yTR,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20150106-092613-B9fc4yTR,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=05 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-32-sp1-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:7c:7d:7e:7f:80,id=idSzpSJQ,vectors=4,netdev=idHNNg5w,bus=pci.0,addr=06  \
    -netdev tap,id=idHNNg5w,vhost=on,vhostfd=23,fd=22  \
    -m 8192  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'SandyBridge',hv_relaxed,hv_relaxed \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/ISO/Windows7/en_windows_7_ultimate_with_sp1_x86_dvd_u_677460.iso \
    -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -drive id=drive_winutils,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-drive,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 \
    -drive id=drive_unattended,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-32-sp1/autounattend.iso \
    -device ide-drive,id=unattended,drive=drive_unattended,bus=ide.1,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,password=123456,addr=0,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off \
    -enable-kvm


# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64

# uname -r
2.6.32-519.el6.x86_64

virtio-win version:
virtio-win-1.7.2.iso

Comment 4 Max Reitz 2015-02-03 19:05:35 UTC
Reproduced with qemu-kvm-0.12.1.2-2.451.el6 and upstream (007c99fd0fb4e0f0579872bb71f5de99b5943dc2): Loading of the virtio-scsi driver hangs with -device virtio-scsi-pci,...,vectors<4 (-M rhel6.4.0 sets vectors=2; vectors=0 works).

Works fine:

$ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04,vectors=4 -drive id=drive_cd1,if=none,media=cdrom,file=/home/maxx/tmp/en_windows_7_professional_n_with_sp1_x64_dvd_u_677207.iso -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_virtio,if=none,media=cdrom,file=/home/maxx/tmp/virtio-win-0.1-100.iso -device ide-drive,id=winutils,drive=drive_virtio,bus=ide.0,unit=1

Does not work:

$ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04,vectors=2 -drive id=drive_cd1,if=none,media=cdrom,file=/home/maxx/tmp/en_windows_7_professional_n_with_sp1_x64_dvd_u_677207.iso -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_virtio,if=none,media=cdrom,file=/home/maxx/tmp/virtio-win-0.1-100.iso -device ide-drive,id=winutils,drive=drive_virtio,bus=ide.0,unit=1

So apparently this is unrelated to the number of CD-ROM drives and also unrelated to whether there actually is a SCSI drive or not (in this case, there isn't).

Comment 5 Max Reitz 2015-02-03 22:01:35 UTC
It appears that the Windows driver does not use MSI in the case of vectors == 0; in the case of vectors >= 4, it first writes 1, then 2 and finally 3 to the MSI-X vector field (both the configuration and the queue vector). If vectors == 1, the fields are not written to at all; if vectors == 2, 1 is written; and if vectors == 3, 1 and 2 are written.

I have yet to understand basically everything about MSI, but I do know that if the driver writes 3 to the fields, everything works (which might indicate that the driver is actually expecting to write 3 there but fails to do some in some cases; but it might just as well not).

Disabling MSI in qemu "fixes" the problem.

Finally, there is a FIXME in the Windows driver code, which might be related to this issue. In case num_queues + 3 > msix_vectors, num_queues is set to 1 (with an annotated FIXME).

The "FIXME" indicates to me that num_queues + 3 > msix_vectors is not something that is expected; therefore num_queues is set to a minimal value (1) to fix that condition. However, the condition still holds true for msix_vectors < 1 + 3 == 4, which is exactly the condition on which this bug appears.

This FIXME block is directly followed by a rather long conditional block (which seems to initialize the request queues) which is only entered if msix_vectors >= num_queues + 3; which, again, will never happen if msix_vectors < 4.

Therefore, it looks to me like the Windows driver code requires 4 msix_vectors to work, and will not initialize the request queues otherwise (which will then result in the driver hanging).

I am not sure how to solve this, not least because I am not involved with virtio, SCSI and much less with MSI.

The most confusing thing to me is that "vectors" is deliberately set to 2 for -M rhel6.4.0. Judging from the Windows driver code, I had guessed it used one MSI for the control queue, one for the event queue and one per request queue, and that it would not use MSI 0 for some reason. This would then mean that less than 4 (or maybe 3) vectors are simply invalid; but this is directly contradicted by the "vectors" value set for -M rhel6.4.0.

So maybe it just means that the Windows driver is buggy and should be content with just two MSI vectors; and if it cannot work with that, just not use MSI.

I will think about it. Help is greatly appreciated, of course.

Max

Comment 9 Peixiu Hou 2015-12-30 03:06:45 UTC
Created attachment 1110435 [details]
win7-64 installation screenshot

Hello,

Reproduced this issue on vitio-win-1.7.2. 

Verified this issue on version virtio-win-1-7-5. Steps same as comment #0. Install os with '-M rhel6.4.0' and virtio-scsi and 3 cdroms.
The issue can be duplicated. I also tried with '-M pc', install os is succesful. Detail please refer to the attached picture "win7-64 installation screenshot".

kernel 2.6.32-595.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.482.el6.x86_64
virtio-win-1-7-5.iso


Best Regards!
Peixiu Hou

Comment 11 Ladi Prosek 2016-07-11 16:22:33 UTC
If (0 < msix_vectors < num_queues + 3), the driver should just use one MSI vector for all queues or fully fall back to legacy interrupts. The former should be doable, not sure about the latter.

Right now we perform the fallback on the virtio side, configuring queues with VIRTIO_MSI_NO_VECTOR, but there's nothing that would tell Windows that we want to assign an IRQ to the device.

Comment 13 lijin 2016-07-25 06:48:49 UTC
This bug can be reproduced on rhel7.3.

Move this bug to rhel7 product as dev and qe have an agreement that we only fix virtio-win bugs on rhel7 release,rhel6 will be the same as latest rhel7 virtio-win package.

Comment 15 lijin 2016-07-28 06:49:43 UTC
Try this issue on virtio-win-prewhql-123 build
steps same as comment #0

Actual Results:
1.when install guest with "-M rhel6.4.0",after load scsi driver it bsod with code 0x0000000A(IRQL_NOT_LESS_OR_EQUAL)
2.when install guest with "-M rhel6.5.0",guest can be installed correctly.

Based on above ,this issue has NOT been fixed.


qemu cli:
/usr/libexec/qemu-kvm -name virt-tests-vm1 -M rhel6.4.0 -nodefaults -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 -drive id=drive_image1,if=none,aio=native,cache=none,file=win7-64-sp1-virtio.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -m 4096 -smp 14,maxcpus=14,cores=1,threads=2,sockets=7 -drive id=drive_cd1,if=none,aio=native,cache=none,media=cdrom,file=en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_winutils,if=none,cache=none,aio=native,media=cdrom,file=/usr/share/virtio-win/virtio-win.iso -device ide-drive,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 -drive id=drive_unattended,if=none,aio=native,cache=none,media=cdrom,file=virtio-win-prewhql-123.iso -device ide-drive,id=unattended,drive=drive_unattended,bus=ide.1,unit=0 -rtc base=localtime,clock=host,driftfix=none -boot order=cdn,once=d,menu=off -usb -device usb-tablet -vga cirrus -vnc :2 -enable-kvm

packages info:
kernel-3.10.0-472.el7.x86_64
qemu-kvm-rhev-2.6.0-14.el7.x86_64
seabios-1.9.1-4.el7.x86_64

Comment 16 Ladi Prosek 2016-07-28 07:36:11 UTC
Thank you, I'm setting up this repro on my host and will report back.

Comment 17 Ladi Prosek 2016-07-28 09:17:31 UTC
OK, there's one more place where we assume that each queue has its MSI vector. Fix coming.

05 fffff880`0390eb90 fffff800`0d37a9ac nt!KiPageFault+0x260
06 fffff880`0390ed20 fffff880`0141045f nt!KeAcquireInterruptSpinLock+0x1c
07 fffff880`0390ed70 fffff880`01417754 storport!StorpAcquireMSISpinLock+0x6f
08 fffff880`0390eda0 fffff880`045936c0 storport!StorPortExtendedFunction+0x194
09 (Inline Function) --------`-------- vioscsi!StorPortAcquireMSISpinLock+0x15 [c:\program files (x86)\windows kits\10\include\10.0.10586.0\km\storport.h @ 10029]
0a fffff880`0390ee20 fffff880`0459348d vioscsi!VioScsiVQLock+0x48 [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\vioscsi\helper.c @ 451]
0b fffff880`0390ee60 fffff880`04594cbb vioscsi!SendSRB+0xc9 [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\vioscsi\helper.c @ 71]
0c fffff880`0390ef20 fffff880`01402269 vioscsi!VioScsiStartIo+0x3b [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\vioscsi\vioscsi.c @ 799]
0d fffff880`0390ef60 fffff800`0d20f62f storport!RaidpAdapterContinueScatterGather+0x149

Comment 19 lijin 2016-08-01 05:32:44 UTC
try with build 124,guest can be installed correctly.

So this issue has been fixed,thank you Ladi.

Change status to verified.

Comment 21 errata-xmlrpc 2016-11-04 08:43:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2609.html


Note You need to log in before you can comment on or make changes to this bug.