Bug 1296531

Summary: [virtio-win][netkvm][rhel6]win8.1 guest bsod with BUGCODE_NDIS_DRIVER(7c) when disable netkvm driver after uninstall/install driver while running netperf test in guest
Product: Red Hat Enterprise Linux 7 Reporter: wehao <wehao>
Component: virtio-winAssignee: Yvugenfi <yvugenfi>
virtio-win sub component: distribution QA Contact: Virtualization Bugs <virt-bugs>
Status: CLOSED WORKSFORME Docs Contact:
Severity: high    
Priority: high CC: lijin, nobody, wehao, wyu, yvugenfi
Version: 7.3   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-23 11:24:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description wehao 2016-01-07 13:02:51 UTC
Description of problem:
while running netperf test in guest, disable netkvm driver in device manager, then BSOD.

Version-Release number of selected component (if applicable):
virtio-win-1.7.5-0.el6.noarch
qemu-kvm-rhev-0.12.1.2-2.482.el6.x86_64
kernel-2.6.32-595.el6.x86_64

How reproducible:
it seems that BSOD appears at the first time executing disable operation on a new guest, it won't show again after rebooting caused by BSOD.

Steps to Reproduce:
1. run netperf test in guest(guest run as netperf client)
   for %i in (32 64 128 256 512 1024 2048 4096 8192 16384 32768) do for %j in (TCP_STREAM,UDP_STREAM) do netperf.exe -H netper_server_ip -c -C -t %j -f m -l 180 -- -m %i done done
2. during step1,disable netkvm driver in device manager

Actual results:
BSOD(7c)


Expected results:
"Redhat VirtIO Ethernet Adapter" is in enabled status in device manager


Additional info:
Maybe BSOD appears on the premise of uninstalling and upgrading the netkvm driver at first.

Comment 3 wehao 2016-01-08 07:56:44 UTC
Additional Info:

1.boot guest with:
/usr/libexec/qemu-kvm \
-name win8-32 \
-enable-kvm -m 3G -smp 4 -uuid 2fab21cf-fd2f-40f7-aba8-c9485c578aff \
-nodefconfig --nodefaults -boot order=cd,menu=on  -drive file=/home/netkvm/win.raw,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/home/netkvm/en_windows_8_1_enterprise_x86_dvd_2972289.iso,media=cdrom,id=cdrom,if=none -device ide-drive,drive=cdrom,bootindex=1 -cdrom /usr/share/virtio-win/virtio-win.iso -vnc 0.0.0.0:1 -vga cirrus -monitor stdio -qmp tcp:0:4449,server,nowait -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=1a:46:0b:ca:bc:7C 

2.Steps to Reproduce:
  1) run netperf test in guest(guest run as netperf client)
   for %i in (32 64 128 256 512 1024 2048 4096 8192 16384 32768) do for %j in   (TCP_STREAM,UDP_STREAM) do netperf.exe -H netper_server_ip -c -C -t %j -f m -l 180 -- -m %i done done
  2) during step1,uninstall netkvm driver in device manager
  3) upgrade netkvm driver in device manager
  4)run netperf test in guest(guest run as netperf client)
   for %i in (32 64 128 256 512 1024 2048 4096 8192 16384 32768) do for %j in   (TCP_STREAM,UDP_STREAM) do netperf.exe -H netper_server_ip -c -C -t %j -f m -l 180 -- -m %i done done
  5)during step4,disable netkvm driver in device manager

3.
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

BUGCODE_NDIS_DRIVER (7c)
This is the NDIS Driver Bugcheck for Windows Server 2003 and later.
For Windows 2000 and Windows XP, see 0xD2, BUGCODE_ID_DRIVER.
Arguments:
Arg1: 0000001f, NDIS BugCheck Code
Arg2: 9a2610e8
Arg3: 00000001
Arg4: 00000000

Debugging Details:
------------------

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!KPRCB                                      ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!KPRCB                                      ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************

ADDITIONAL_DEBUG_TEXT:  
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

MODULE_NAME: ndis

FAULTING_MODULE: 8181b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  523d44b4

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  0x7C

CURRENT_IRQL:  0

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

LAST_CONTROL_TRANSFER:  from 82ef3ec4 to 8191b5d4

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
8742567c 82ef3ec4 0000007c 0000001f 9a2610e8 nt!KeBugCheckEx
8742569c 82f67465 00000000 82f263d0 9a2610e8 ndis!NdisFreeBufferPool+0x545a
874256d8 82f39a42 b0f298c0 9a261da0 00000000 ndis!NdisMSetAttributesEx+0x918
87425868 82f34bd9 8cdefd88 9a261da0 9a261da0 ndis!NdisMDeregisterMiniportDriver+0xe72
87425884 82f3bc16 9a261da0 874258b4 82f390d6 ndis!NdisMSetMiniportAttributes+0x302f
87425890 82f390d6 00000000 874258a4 9a2610e8 ndis!NdisCloseAdapterEx+0x199e
874258b4 82f645ad 00000000 00000002 9a2610e8 ndis!NdisMDeregisterMiniportDriver+0x506
874259e0 82f38eb8 9a2610e8 00000000 81861360 ndis!NdisSetAoAcOptions+0x4228
87425a18 82f354da 890a6e90 87425a3e 87425a3f ndis!NdisMDeregisterMiniportDriver+0x2e8
87425a40 80623808 9a261030 890a6e90 86e4f3f8 ndis!NdisMSetMiniportAttributes+0x3930
87425a6c 81cdf635 9a261030 890a6e90 82f35379 VerifierExt!XdvHibernationNotification+0x97dc
87425a80 81cdf933 80623738 0000001b 87425ab4 nt!IoIsValidIrpStatus+0x14160
87425a90 81cc8b4d 9a261030 890a6e90 81cc8881 nt!IoIsValidIrpStatus+0x1445e
87425ab4 8185fb22 81cdfe61 890a6fac 9a261030 nt!MmIsDriverSuspectForVerifier+0x1f5d
87425ac8 81cdfe61 86e59c98 890a6e90 ae6fa958 nt!IofCallDriver+0x62
87425ae0 81cc8b4d ae6fa958 9a261030 81cc8881 nt!IoIsValidIrpStatus+0x1498c
87425b04 8185fb22 81ac0e8e 890a6fd8 87425b9c nt!MmIsDriverSuspectForVerifier+0x1f5d
87425b18 81ac0e8e 88b8b030 88b8c4a0 00000002 nt!IofCallDriver+0x62
87425b54 81af9a39 c00000bb 00000000 00000000 nt!FsRtlSetKernelEaFile+0x1a8
87425bb0 818b1a91 00000000 88b8c4a0 88b8c401 nt!FsRtlTeardownPerStreamContexts+0x30a3
87425be4 81afb8fa b1ad3dd8 00000000 88b8c4a0 nt!ExFreeCacheAwareRundownProtection+0x25d
87425bf8 81afb89a 00000016 b1ad3dd8 00000000 nt!FsRtlTeardownPerStreamContexts+0x4f64
87425c34 81afacbd 00000002 00000001 00000016 nt!FsRtlTeardownPerStreamContexts+0x4f04
87425cd8 81af250a 00000000 88f0bc28 00000001 nt!FsRtlTeardownPerStreamContexts+0x4327
87425cf8 81aeefab 81aeed10 84f84700 89df0250 nt!PoDeletePowerRequest+0x94
87425d24 818a7611 89df0250 00000000 84f84700 nt!RtlGetOwnerSecurityDescriptor+0xe4d
87425d70 818b577a 00000000 e674b80f 00000000 nt!RtlFindSetBits+0xa63
87425db0 81930fe1 818a7512 00000000 00000000 nt!MmIsDriverVerifying+0x7e
87425db4 818a7512 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x741
87425db8 00000000 00000000 00000000 19191919 nt!RtlFindSetBits+0x964


STACK_COMMAND:  kb

FOLLOWUP_IP: 
ndis!NdisFreeBufferPool+545a
82ef3ec4 57              push    edi

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  ndis!NdisFreeBufferPool+545a

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ndis.sys

BUCKET_ID:  WRONG_SYMBOLS

FAILURE_BUCKET_ID:  WRONG_SYMBOLS

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:wrong_symbols

FAILURE_ID_HASH:  {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner

Comment 4 lijin 2016-01-11 03:20:30 UTC
please try with virtio-win-1.7.4-1.el6_7.2 to check whether it's a regression bug

Comment 5 wehao 2016-01-12 06:07:35 UTC
(In reply to lijin from comment #4)
> please try with virtio-win-1.7.4-1.el6_7.2 to check whether it's a
> regression bug

It didn't reproduce.

Comment 6 wehao 2016-01-12 06:08:10 UTC
(In reply to lijin from comment #4)
> please try with virtio-win-1.7.4-1.el6_7.2 to check whether it's a
> regression bug

It didn't reproduce on virtio-win-1.7.4-1.el6_7.2

Comment 8 Yvugenfi@redhat.com 2016-03-21 14:17:58 UTC
Hello,

I cannot reproduce the issue with the mentioned versions of QEMU and virti-win drivers.

Can you please clarify what you mean by:

"Additional info:
Maybe BSOD appears on the premise of uninstalling and upgrading the netkvm driver at first."

Best regards,
Yan.

Comment 9 Yu Wang 2016-03-22 02:26:52 UTC
Hi,

wehao only can reproduce it when it is a clear OS and install the driver first time. And not hit this issue everytime.

Thanks
wyu

Comment 10 Yvugenfi@redhat.com 2016-03-22 09:09:57 UTC
Can you please clarify more on the steps to reproduce (I tried to reproduce after clear OS installation as well but failed).

For example - do you install driver right after install process, do you reboot in between or maybe there are some other actions?

Thanks.

Comment 11 Yu Wang 2016-03-22 09:37:54 UTC
Hi,

Steps as following:
1 install the netkvm driver on a clean OS and successfully installed

2 run the case of uninstalling driver during running netperf 

3 run the case of upgrading the netkvm driver 

4 run netperf , command as comment#0

5 during step2, disable the netkvm driver, then BSOD occurred.


Additional info:

If it has occurred  BSOD once , even re-run disable netkvm driver during netperf, it will not occurred BSOD on this guest any more.

Comment 12 Yvugenfi@redhat.com 2016-03-22 11:02:13 UTC
Can you please specify what version is installed first and what version do you upgrade to?

Thanks!

Comment 13 Yu Wang 2016-03-23 02:06:37 UTC
Hi Yan,

Since wehao has left Redhat, you can needinfo me(wyu) for further development. I will communicate with her for the details.

Sorry, wehao cannot remember what version she upgraded. But she installed virtio-win-1.7.5-0.el6.noarch first ,then upgrade to a version(wehao cannot remember,may be virtio-win-prewhql-112) and downgrade back to virtio-win-1.7.5-0.el6.noarch.


Thanks
wyu

Comment 14 Yvugenfi@redhat.com 2016-03-23 09:05:20 UTC
Hi Wyu,


Thanks!

Can you please try and reproduce this bug with win-1.7.5-0.el6.noarch and latest virtio-win-prewhql build?

Thanks,
Yan.

Comment 15 Yu Wang 2016-03-23 09:40:33 UTC
(In reply to Yan Vugenfirer from comment #14)
> Hi Wyu,
> 
> 
> Thanks!
> 
> Can you please try and reproduce this bug with win-1.7.5-0.el6.noarch and
> latest virtio-win-prewhql build?
> 
> Thanks,
> Yan.

Yes, I will add comment when I have tried.

Comment 16 Yu Wang 2016-03-23 10:20:13 UTC
Hi Yan

I tried three times as comment#11 and comment#14 , but I still cannot reproduce this issue.

Thanks
wyu

Comment 17 Yvugenfi@redhat.com 2016-03-23 11:24:46 UTC
Thanks!

I suggest to close the bug for now. Could be an issue with intermediate version that was used for upgrade.