Bug 1157987

Created attachment 951307 [details]
screenshot

Created attachment 976671 [details]
win7-32 screendump

win7-32 met it too, and stuck here.

cmd line:
/usr/bin/qemu-kvm \
    -name 'virt-tests-vm1' \
    -M rhel6.4.0  \
    -nodefaults  \
    -vga qxl \
    -device AC97,bus=pci.0,addr=03  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20150106-092613-B9fc4yTR,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20150106-092613-B9fc4yTR,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20150106-092613-B9fc4yTR,path=/tmp/seabios-20150106-092613-B9fc4yTR,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20150106-092613-B9fc4yTR,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=05 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-32-sp1-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:7c:7d:7e:7f:80,id=idSzpSJQ,vectors=4,netdev=idHNNg5w,bus=pci.0,addr=06  \
    -netdev tap,id=idHNNg5w,vhost=on,vhostfd=23,fd=22  \
    -m 8192  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'SandyBridge',hv_relaxed,hv_relaxed \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/ISO/Windows7/en_windows_7_ultimate_with_sp1_x86_dvd_u_677460.iso \
    -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -drive id=drive_winutils,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-drive,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 \
    -drive id=drive_unattended,if=none,snapshot=off,aio=native,media=cdrom,file=/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-32-sp1/autounattend.iso \
    -device ide-drive,id=unattended,drive=drive_unattended,bus=ide.1,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,password=123456,addr=0,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=d,menu=off,strict=off \
    -enable-kvm


# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64

# uname -r
2.6.32-519.el6.x86_64

virtio-win version:
virtio-win-1.7.2.iso

Reproduced with qemu-kvm-0.12.1.2-2.451.el6 and upstream (007c99fd0fb4e0f0579872bb71f5de99b5943dc2): Loading of the virtio-scsi driver hangs with -device virtio-scsi-pci,...,vectors<4 (-M rhel6.4.0 sets vectors=2; vectors=0 works).

Works fine:

$ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04,vectors=4 -drive id=drive_cd1,if=none,media=cdrom,file=/home/maxx/tmp/en_windows_7_professional_n_with_sp1_x64_dvd_u_677207.iso -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_virtio,if=none,media=cdrom,file=/home/maxx/tmp/virtio-win-0.1-100.iso -device ide-drive,id=winutils,drive=drive_virtio,bus=ide.0,unit=1

Does not work:

$ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 512 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04,vectors=2 -drive id=drive_cd1,if=none,media=cdrom,file=/home/maxx/tmp/en_windows_7_professional_n_with_sp1_x64_dvd_u_677207.iso -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_virtio,if=none,media=cdrom,file=/home/maxx/tmp/virtio-win-0.1-100.iso -device ide-drive,id=winutils,drive=drive_virtio,bus=ide.0,unit=1

So apparently this is unrelated to the number of CD-ROM drives and also unrelated to whether there actually is a SCSI drive or not (in this case, there isn't).

It appears that the Windows driver does not use MSI in the case of vectors == 0; in the case of vectors >= 4, it first writes 1, then 2 and finally 3 to the MSI-X vector field (both the configuration and the queue vector). If vectors == 1, the fields are not written to at all; if vectors == 2, 1 is written; and if vectors == 3, 1 and 2 are written.

I have yet to understand basically everything about MSI, but I do know that if the driver writes 3 to the fields, everything works (which might indicate that the driver is actually expecting to write 3 there but fails to do some in some cases; but it might just as well not).

Disabling MSI in qemu "fixes" the problem.

Finally, there is a FIXME in the Windows driver code, which might be related to this issue. In case num_queues + 3 > msix_vectors, num_queues is set to 1 (with an annotated FIXME).

The "FIXME" indicates to me that num_queues + 3 > msix_vectors is not something that is expected; therefore num_queues is set to a minimal value (1) to fix that condition. However, the condition still holds true for msix_vectors < 1 + 3 == 4, which is exactly the condition on which this bug appears.

This FIXME block is directly followed by a rather long conditional block (which seems to initialize the request queues) which is only entered if msix_vectors >= num_queues + 3; which, again, will never happen if msix_vectors < 4.

Therefore, it looks to me like the Windows driver code requires 4 msix_vectors to work, and will not initialize the request queues otherwise (which will then result in the driver hanging).

I am not sure how to solve this, not least because I am not involved with virtio, SCSI and much less with MSI.

The most confusing thing to me is that "vectors" is deliberately set to 2 for -M rhel6.4.0. Judging from the Windows driver code, I had guessed it used one MSI for the control queue, one for the event queue and one per request queue, and that it would not use MSI 0 for some reason. This would then mean that less than 4 (or maybe 3) vectors are simply invalid; but this is directly contradicted by the "vectors" value set for -M rhel6.4.0.

So maybe it just means that the Windows driver is buggy and should be content with just two MSI vectors; and if it cannot work with that, just not use MSI.

I will think about it. Help is greatly appreciated, of course.

Max

Created attachment 1110435 [details]
win7-64 installation screenshot

Hello,

Reproduced this issue on vitio-win-1.7.2. 

Verified this issue on version virtio-win-1-7-5. Steps same as comment #0. Install os with '-M rhel6.4.0' and virtio-scsi and 3 cdroms.
The issue can be duplicated. I also tried with '-M pc', install os is succesful. Detail please refer to the attached picture "win7-64 installation screenshot".

kernel 2.6.32-595.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.482.el6.x86_64
virtio-win-1-7-5.iso


Best Regards!
Peixiu Hou

If (0 < msix_vectors < num_queues + 3), the driver should just use one MSI vector for all queues or fully fall back to legacy interrupts. The former should be doable, not sure about the latter.

Right now we perform the fallback on the virtio side, configuring queues with VIRTIO_MSI_NO_VECTOR, but there's nothing that would tell Windows that we want to assign an IRQ to the device.

This bug can be reproduced on rhel7.3.

Move this bug to rhel7 product as dev and qe have an agreement that we only fix virtio-win bugs on rhel7 release,rhel6 will be the same as latest rhel7 virtio-win package.

Try this issue on virtio-win-prewhql-123 build
steps same as comment #0

Actual Results:
1.when install guest with "-M rhel6.4.0",after load scsi driver it bsod with code 0x0000000A(IRQL_NOT_LESS_OR_EQUAL)
2.when install guest with "-M rhel6.5.0",guest can be installed correctly.

Based on above ,this issue has NOT been fixed.


qemu cli:
/usr/libexec/qemu-kvm -name virt-tests-vm1 -M rhel6.4.0 -nodefaults -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 -drive id=drive_image1,if=none,aio=native,cache=none,file=win7-64-sp1-virtio.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -m 4096 -smp 14,maxcpus=14,cores=1,threads=2,sockets=7 -drive id=drive_cd1,if=none,aio=native,cache=none,media=cdrom,file=en_windows_7_ultimate_with_sp1_x64_dvd_u_677332.iso -device ide-drive,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 -drive id=drive_winutils,if=none,cache=none,aio=native,media=cdrom,file=/usr/share/virtio-win/virtio-win.iso -device ide-drive,id=winutils,drive=drive_winutils,bus=ide.0,unit=1 -drive id=drive_unattended,if=none,aio=native,cache=none,media=cdrom,file=virtio-win-prewhql-123.iso -device ide-drive,id=unattended,drive=drive_unattended,bus=ide.1,unit=0 -rtc base=localtime,clock=host,driftfix=none -boot order=cdn,once=d,menu=off -usb -device usb-tablet -vga cirrus -vnc :2 -enable-kvm

packages info:
kernel-3.10.0-472.el7.x86_64
qemu-kvm-rhev-2.6.0-14.el7.x86_64
seabios-1.9.1-4.el7.x86_64

Thank you, I'm setting up this repro on my host and will report back.

OK, there's one more place where we assume that each queue has its MSI vector. Fix coming.

05 fffff880`0390eb90 fffff800`0d37a9ac nt!KiPageFault+0x260
06 fffff880`0390ed20 fffff880`0141045f nt!KeAcquireInterruptSpinLock+0x1c
07 fffff880`0390ed70 fffff880`01417754 storport!StorpAcquireMSISpinLock+0x6f
08 fffff880`0390eda0 fffff880`045936c0 storport!StorPortExtendedFunction+0x194
09 (Inline Function) --------`-------- vioscsi!StorPortAcquireMSISpinLock+0x15 [c:\program files (x86)\windows kits\10\include\10.0.10586.0\km\storport.h @ 10029]
0a fffff880`0390ee20 fffff880`0459348d vioscsi!VioScsiVQLock+0x48 [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\vioscsi\helper.c @ 451]
0b fffff880`0390ee60 fffff880`04594cbb vioscsi!SendSRB+0xc9 [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\vioscsi\helper.c @ 71]
0c fffff880`0390ef20 fffff880`01402269 vioscsi!VioScsiStartIo+0x3b [c:\cygwin64\tmp\build\source\internal-kvm-guest-drivers-windows\vioscsi\vioscsi.c @ 799]
0d fffff880`0390ef60 fffff800`0d20f62f storport!RaidpAdapterContinueScatterGather+0x149

try with build 124,guest can be installed correctly.

So this issue has been fixed,thank you Ladi.

Change status to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2609.html