Bug 1352432 - [virtio-win][vioscsi]Win2012-64&R2 guest occurred bsod(d1) when whql test DPWLK- Hot-Replace - Device Test - Verify driver support for D3 power state
Summary: [virtio-win][vioscsi]Win2012-64&R2 guest occurred bsod(d1) when whql test DPW...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virtio-win
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Ladi Prosek
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-04 07:39 UTC by Peixiu Hou
Modified: 2016-11-04 08:54 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
NO_DOCS
Clone Of:
Environment:
Last Closed: 2016-11-04 08:54:55 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2609 normal SHIPPED_LIVE virtio-win bug fix and enhancement update 2016-11-03 15:27:12 UTC

Description Peixiu Hou 2016-07-04 07:39:18 UTC
Description of problem:
win2012-64&R2 guest bsod(d1) when whql test whql test "DPWLK- Hot-Replace - Device Test - Verify driver support for D3 power state" job

Version-Release number of selected component (if applicable):
kernel-3.10.0-454.el7.x86_64
qemu-kvm-rhev-2.6.0-10.el7.x86_64
virtio-win-prewhql-121

How reproducible:
100%

Steps to Reproduce:
1. Boot guest with:
/usr/libexec/qemu-kvm -name 121SCS2012R2CMH -enable-kvm -m 6G -smp 8 -uuid b9392453-00af-4f28-9beb-6c3b549f4e18 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/tmp/121SCS2012R2CMH,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=121SCS2012R2CMH,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=en_windows_server_2012_r2_x64_dvd_2707946.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=121SCS2012R2CMH.vfd,if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:52:18:70:f6:86,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:1 -vga cirrus -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x7,num_queues=8,disable-legacy=on,disable-modern=off -drive file=121SCS2012R2CMH_test.raw,if=none,id=drive-scsi-disk0,format=raw,serial=mike_cao,cache=none -device scsi-hd,bus=scsi0.0,drive=drive-scsi-disk0,id=scsi-disk0

2. Run whql test DPWLK-Hot-Replace - Device Test - Verify driver support for D3 power state job

Actual results:
bsod(D1)

Expected results:
Pass

Additional info:
1. BSOD(D1) occurred w/o virtio-1.0

2. Tried with virtio-win-prewhql-120, issue can be reproduced.

3. The BSOD dump file has uploaded follow location:
http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/bug1234751/

4. kd> ! analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000000, memory referenced
Arg2: 0000000000000009, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff800009a3345, address which referenced memory

Debugging Details:
------------------

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!KPRCB                                      ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!KPRCB                                      ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************

ADDITIONAL_DEBUG_TEXT:  
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

MODULE_NAME: vioscsi

FAULTING_MODULE: fffff80341c77000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  577250b1

READ_ADDRESS:  0000000000000000 

CURRENT_IRQL:  0

FAULTING_IP: 
vioscsi+4345
fffff800`009a3345 483929          cmp     qword ptr [rcx],rbp

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  AV

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

LAST_CONTROL_TRANSFER:  from fffff80341dd2be9 to fffff80341dc70a0

STACK_TEXT:  
ffffd000`20a7f4b8 fffff803`41dd2be9 : 00000000`0000000a 00000000`00000000 00000000`00000009 00000000`00000000 : nt!KeBugCheckEx
ffffd000`20a7f4c0 fffff803`41dd143a : 00000000`00000000 00000000`00000000 ffffe000`011ad200 ffffd000`20a7f600 : nt!setjmpex+0x37e9
ffffd000`20a7f600 fffff800`009a3345 : ffffe000`011ad260 00000000`00000100 ffffe000`00c72c10 fffff803`422e0470 : nt!setjmpex+0x203a
ffffd000`20a7f790 fffff800`009a058a : ffffcf80`00dc8c02 ffffe000`011b1010 ffffd000`2204a010 ffffd000`2204a400 : vioscsi+0x4345
ffffd000`20a7f7c0 fffff800`009a1047 : fffff800`00838010 00000000`00000000 00000000`00000000 fffff800`0080ad10 : vioscsi+0x158a
ffffd000`20a7f7f0 fffff800`00809446 : ffffcf80`00dc8e50 ffffe000`011ac1a0 ffffcf80`00dc8e50 fffff803`422db33d : vioscsi+0x2047
ffffd000`20a7f830 fffff800`0080ab84 : ffffe000`011ac1a0 ffffcf80`00dc8cf0 00000000`00000000 fffff803`41d901c6 : storport!StorPortStallExecution+0x36
ffffd000`20a7f860 fffff800`0080aa1e : ffffe000`011ac1a0 ffffcf80`00dc8cf0 ffffe000`0017a501 fffff803`41cba118 : storport!StorPortStallExecution+0x1774
ffffd000`20a7f890 fffff800`008048d3 : ffffe000`011ac1a0 00000000`00000000 ffffd000`00000001 ffffd000`00000000 : storport!StorPortStallExecution+0x160e
ffffd000`20a7f8f0 fffff803`41cfae90 : ffffd000`20a7fad0 fffff803`41cbc266 ffffd000`20a7fc70 ffffe000`011ac118 : storport!StorPortGetScatterGatherList+0xa33
ffffd000`20a7f9d0 fffff803`41cfa111 : ffffd000`20a7fc20 ffffd000`20a7fc20 ffffe000`001779f0 ffffd000`20a7fc60 : nt!NtTraceEvent+0x1ea0
ffffd000`20a7fb20 fffff803`41dcabea : ffffd000`20a55180 ffffd000`20a55180 ffffd000`20a61200 ffffe000`05e59680 : nt!NtTraceEvent+0x1121
ffffd000`20a7fda0 00000000`00000000 : ffffd000`20a80000 ffffd000`20a7a000 00000000`00000000 00000000`00000000 : nt!KeSynchronizeExecution+0x2efa


STACK_COMMAND:  kb

FOLLOWUP_IP: 
vioscsi+4345
fffff800`009a3345 483929          cmp     qword ptr [rcx],rbp

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  vioscsi+4345

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  vioscsi.sys

BUCKET_ID:  WRONG_SYMBOLS

FAILURE_BUCKET_ID:  WRONG_SYMBOLS

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:wrong_symbols

FAILURE_ID_HASH:  {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner
---------

Comment 3 Vadim Rozenfeld 2016-07-12 08:21:16 UTC
Should bi fixed in build 122
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=503003

Comment 5 Vadim Rozenfeld 2016-07-13 05:09:17 UTC
Unfortunately I still can reproduce this issue with the latest drivers from build 122.

Comment 6 Vadim Rozenfeld 2016-07-26 09:20:03 UTC
Does the problem exist with single-queue configuration?
Another question regarding to the testing configuration,
do you run this on a system or on a date disk?

Thanks,
Vadim.

Comment 7 Ladi Prosek 2016-07-26 13:27:15 UTC
I can reproduce a reliable BSOD with a multiqueue & data disk configuration. It's an attempt to invoke a NULL DPC:

 # Child-SP          RetAddr           Call Site
00 fffff803`1495abd8 fffff803`13087e90 0x0
01 fffff803`1495abe0 fffff803`13087111 nt!KiExecuteAllDpcs+0x1b0
02 fffff803`1495ad30 fffff803`131571d5 nt!KiRetireDpcList+0xe1
03 fffff803`1495afb0 fffff803`13156fd9 nt!KxRetireDpcList+0x5
04 ffffd000`239fe270 fffff803`13159245 nt!KiDispatchInterruptContinue
05 ffffd000`239fe2a0 fffff803`13155987 nt!KiDpcInterruptBypass+0x25
06 ffffd000`239fe2b0 fffff803`130ace2b nt!KiInterruptDispatchLBControl+0x197
07 ffffd000`239fe440 fffff803`130ac878 nt!MmPropagateDirtyBitsToPfn+0xbf
08 ffffd000`239fe480 fffff803`13055d08 nt!CcUnmapVacbArray+0x158
09 ffffd000`239fe4f0 fffff803`13051f6f nt!CcGetVirtualAddress+0x438
0a ffffd000`239fe590 fffff803`1305156b nt!CcMapAndCopyInToCache+0x157
0b ffffd000`239fe680 fffff800`00da377d nt!CcCopyWriteEx+0x1bb
0c ffffd000`239fe720 fffff800`00a795c0 Ntfs!NtfsCopyWriteA+0x2ed
0d ffffd000`239fe9a0 fffff800`00aa027a fltmgr!FltpPerformFastIoCall+0x1f0
0e ffffd000`239fea00 fffff803`133b4943 fltmgr!FltpFastIoWrite+0x1ca
0f ffffd000`239feab0 fffff803`1315f8b3 nt!NtWriteFile+0x5c7
10 ffffd000`239febd0 00007ffe`f5387b5a nt!KiSystemServiceCopyEnd+0x13

It does appear to be a regression, I'm bisecting now.

Comment 8 Peixiu Hou 2016-08-02 06:36:00 UTC
Verified this bug with virtio-win-prewhql-124,

Steps as comment#0

Job "DPWLK- Hot-Replace - Device Test - Verify driver support for D3 power state" passed on win2012 and win2012R2. The bug has been fixed, thanks~


Best Regards~
Peixiu Hou

Comment 9 lijin 2016-08-02 06:43:51 UTC
change status to verified according to comment#8

Comment 12 errata-xmlrpc 2016-11-04 08:54:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2609.html


Note You need to log in before you can comment on or make changes to this bug.