Bug 1296531 - [virtio-win][netkvm][rhel6]win8.1 guest bsod with BUGCODE_NDIS_DRIVER(7c) when disable netkvm driver after uninstall/install driver while running netperf test in guest
[virtio-win][netkvm][rhel6]win8.1 guest bsod with BUGCODE_NDIS_DRIVER(7c) whe...
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virtio-win (Show other bugs)
7.3
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Yan Vugenfirer
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-07 08:02 EST by wehao
Modified: 2016-09-19 21:37 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-23 07:24:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description wehao 2016-01-07 08:02:51 EST
Description of problem:
while running netperf test in guest, disable netkvm driver in device manager, then BSOD.

Version-Release number of selected component (if applicable):
virtio-win-1.7.5-0.el6.noarch
qemu-kvm-rhev-0.12.1.2-2.482.el6.x86_64
kernel-2.6.32-595.el6.x86_64

How reproducible:
it seems that BSOD appears at the first time executing disable operation on a new guest, it won't show again after rebooting caused by BSOD.

Steps to Reproduce:
1. run netperf test in guest(guest run as netperf client)
   for %i in (32 64 128 256 512 1024 2048 4096 8192 16384 32768) do for %j in (TCP_STREAM,UDP_STREAM) do netperf.exe -H netper_server_ip -c -C -t %j -f m -l 180 -- -m %i done done
2. during step1,disable netkvm driver in device manager

Actual results:
BSOD(7c)


Expected results:
"Redhat VirtIO Ethernet Adapter" is in enabled status in device manager


Additional info:
Maybe BSOD appears on the premise of uninstalling and upgrading the netkvm driver at first.
Comment 3 wehao 2016-01-08 02:56:44 EST
Additional Info:

1.boot guest with:
/usr/libexec/qemu-kvm \
-name win8-32 \
-enable-kvm -m 3G -smp 4 -uuid 2fab21cf-fd2f-40f7-aba8-c9485c578aff \
-nodefconfig --nodefaults -boot order=cd,menu=on  -drive file=/home/netkvm/win.raw,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/home/netkvm/en_windows_8_1_enterprise_x86_dvd_2972289.iso,media=cdrom,id=cdrom,if=none -device ide-drive,drive=cdrom,bootindex=1 -cdrom /usr/share/virtio-win/virtio-win.iso -vnc 0.0.0.0:1 -vga cirrus -monitor stdio -qmp tcp:0:4449,server,nowait -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=1a:46:0b:ca:bc:7C 

2.Steps to Reproduce:
  1) run netperf test in guest(guest run as netperf client)
   for %i in (32 64 128 256 512 1024 2048 4096 8192 16384 32768) do for %j in   (TCP_STREAM,UDP_STREAM) do netperf.exe -H netper_server_ip -c -C -t %j -f m -l 180 -- -m %i done done
  2) during step1,uninstall netkvm driver in device manager
  3) upgrade netkvm driver in device manager
  4)run netperf test in guest(guest run as netperf client)
   for %i in (32 64 128 256 512 1024 2048 4096 8192 16384 32768) do for %j in   (TCP_STREAM,UDP_STREAM) do netperf.exe -H netper_server_ip -c -C -t %j -f m -l 180 -- -m %i done done
  5)during step4,disable netkvm driver in device manager

3.
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

BUGCODE_NDIS_DRIVER (7c)
This is the NDIS Driver Bugcheck for Windows Server 2003 and later.
For Windows 2000 and Windows XP, see 0xD2, BUGCODE_ID_DRIVER.
Arguments:
Arg1: 0000001f, NDIS BugCheck Code
Arg2: 9a2610e8
Arg3: 00000001
Arg4: 00000000

Debugging Details:
------------------

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!KPRCB                                      ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!KPRCB                                      ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
***    doesn't have full symbol information.  Unqualified symbol      ***
***    resolution is turned off by default. Please either specify a   ***
***    fully qualified symbol module!symbolname, or enable resolution ***
***    of unqualified symbols by typing ".symopt- 100". Note that   ***
***    enabling unqualified symbol resolution with network symbol     ***
***    server shares in the symbol path may cause the debugger to     ***
***    appear to hang for long periods of time when an incorrect      ***
***    symbol name is typed or the network symbol server is down.     ***
***                                                                   ***
***    For some commands to work properly, your symbol path           ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************

ADDITIONAL_DEBUG_TEXT:  
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

MODULE_NAME: ndis

FAULTING_MODULE: 8181b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  523d44b4

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  0x7C

CURRENT_IRQL:  0

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

LAST_CONTROL_TRANSFER:  from 82ef3ec4 to 8191b5d4

STACK_TEXT:  
WARNING: Stack unwind information not available. Following frames may be wrong.
8742567c 82ef3ec4 0000007c 0000001f 9a2610e8 nt!KeBugCheckEx
8742569c 82f67465 00000000 82f263d0 9a2610e8 ndis!NdisFreeBufferPool+0x545a
874256d8 82f39a42 b0f298c0 9a261da0 00000000 ndis!NdisMSetAttributesEx+0x918
87425868 82f34bd9 8cdefd88 9a261da0 9a261da0 ndis!NdisMDeregisterMiniportDriver+0xe72
87425884 82f3bc16 9a261da0 874258b4 82f390d6 ndis!NdisMSetMiniportAttributes+0x302f
87425890 82f390d6 00000000 874258a4 9a2610e8 ndis!NdisCloseAdapterEx+0x199e
874258b4 82f645ad 00000000 00000002 9a2610e8 ndis!NdisMDeregisterMiniportDriver+0x506
874259e0 82f38eb8 9a2610e8 00000000 81861360 ndis!NdisSetAoAcOptions+0x4228
87425a18 82f354da 890a6e90 87425a3e 87425a3f ndis!NdisMDeregisterMiniportDriver+0x2e8
87425a40 80623808 9a261030 890a6e90 86e4f3f8 ndis!NdisMSetMiniportAttributes+0x3930
87425a6c 81cdf635 9a261030 890a6e90 82f35379 VerifierExt!XdvHibernationNotification+0x97dc
87425a80 81cdf933 80623738 0000001b 87425ab4 nt!IoIsValidIrpStatus+0x14160
87425a90 81cc8b4d 9a261030 890a6e90 81cc8881 nt!IoIsValidIrpStatus+0x1445e
87425ab4 8185fb22 81cdfe61 890a6fac 9a261030 nt!MmIsDriverSuspectForVerifier+0x1f5d
87425ac8 81cdfe61 86e59c98 890a6e90 ae6fa958 nt!IofCallDriver+0x62
87425ae0 81cc8b4d ae6fa958 9a261030 81cc8881 nt!IoIsValidIrpStatus+0x1498c
87425b04 8185fb22 81ac0e8e 890a6fd8 87425b9c nt!MmIsDriverSuspectForVerifier+0x1f5d
87425b18 81ac0e8e 88b8b030 88b8c4a0 00000002 nt!IofCallDriver+0x62
87425b54 81af9a39 c00000bb 00000000 00000000 nt!FsRtlSetKernelEaFile+0x1a8
87425bb0 818b1a91 00000000 88b8c4a0 88b8c401 nt!FsRtlTeardownPerStreamContexts+0x30a3
87425be4 81afb8fa b1ad3dd8 00000000 88b8c4a0 nt!ExFreeCacheAwareRundownProtection+0x25d
87425bf8 81afb89a 00000016 b1ad3dd8 00000000 nt!FsRtlTeardownPerStreamContexts+0x4f64
87425c34 81afacbd 00000002 00000001 00000016 nt!FsRtlTeardownPerStreamContexts+0x4f04
87425cd8 81af250a 00000000 88f0bc28 00000001 nt!FsRtlTeardownPerStreamContexts+0x4327
87425cf8 81aeefab 81aeed10 84f84700 89df0250 nt!PoDeletePowerRequest+0x94
87425d24 818a7611 89df0250 00000000 84f84700 nt!RtlGetOwnerSecurityDescriptor+0xe4d
87425d70 818b577a 00000000 e674b80f 00000000 nt!RtlFindSetBits+0xa63
87425db0 81930fe1 818a7512 00000000 00000000 nt!MmIsDriverVerifying+0x7e
87425db4 818a7512 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x741
87425db8 00000000 00000000 00000000 19191919 nt!RtlFindSetBits+0x964


STACK_COMMAND:  kb

FOLLOWUP_IP: 
ndis!NdisFreeBufferPool+545a
82ef3ec4 57              push    edi

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  ndis!NdisFreeBufferPool+545a

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ndis.sys

BUCKET_ID:  WRONG_SYMBOLS

FAILURE_BUCKET_ID:  WRONG_SYMBOLS

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:wrong_symbols

FAILURE_ID_HASH:  {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner
Comment 4 lijin 2016-01-10 22:20:30 EST
please try with virtio-win-1.7.4-1.el6_7.2 to check whether it's a regression bug
Comment 5 wehao 2016-01-12 01:07:35 EST
(In reply to lijin from comment #4)
> please try with virtio-win-1.7.4-1.el6_7.2 to check whether it's a
> regression bug

It didn't reproduce.
Comment 6 wehao 2016-01-12 01:08:10 EST
(In reply to lijin from comment #4)
> please try with virtio-win-1.7.4-1.el6_7.2 to check whether it's a
> regression bug

It didn't reproduce on virtio-win-1.7.4-1.el6_7.2
Comment 8 Yan Vugenfirer 2016-03-21 10:17:58 EDT
Hello,

I cannot reproduce the issue with the mentioned versions of QEMU and virti-win drivers.

Can you please clarify what you mean by:

"Additional info:
Maybe BSOD appears on the premise of uninstalling and upgrading the netkvm driver at first."

Best regards,
Yan.
Comment 9 Yu Wang 2016-03-21 22:26:52 EDT
Hi,

wehao only can reproduce it when it is a clear OS and install the driver first time. And not hit this issue everytime.

Thanks
wyu
Comment 10 Yan Vugenfirer 2016-03-22 05:09:57 EDT
Can you please clarify more on the steps to reproduce (I tried to reproduce after clear OS installation as well but failed).

For example - do you install driver right after install process, do you reboot in between or maybe there are some other actions?

Thanks.
Comment 11 Yu Wang 2016-03-22 05:37:54 EDT
Hi,

Steps as following:
1 install the netkvm driver on a clean OS and successfully installed

2 run the case of uninstalling driver during running netperf 

3 run the case of upgrading the netkvm driver 

4 run netperf , command as comment#0

5 during step2, disable the netkvm driver, then BSOD occurred.


Additional info:

If it has occurred  BSOD once , even re-run disable netkvm driver during netperf, it will not occurred BSOD on this guest any more.
Comment 12 Yan Vugenfirer 2016-03-22 07:02:13 EDT
Can you please specify what version is installed first and what version do you upgrade to?

Thanks!
Comment 13 Yu Wang 2016-03-22 22:06:37 EDT
Hi Yan,

Since wehao has left Redhat, you can needinfo me(wyu@redhat.com) for further development. I will communicate with her for the details.

Sorry, wehao cannot remember what version she upgraded. But she installed virtio-win-1.7.5-0.el6.noarch first ,then upgrade to a version(wehao cannot remember,may be virtio-win-prewhql-112) and downgrade back to virtio-win-1.7.5-0.el6.noarch.


Thanks
wyu
Comment 14 Yan Vugenfirer 2016-03-23 05:05:20 EDT
Hi Wyu,


Thanks!

Can you please try and reproduce this bug with win-1.7.5-0.el6.noarch and latest virtio-win-prewhql build?

Thanks,
Yan.
Comment 15 Yu Wang 2016-03-23 05:40:33 EDT
(In reply to Yan Vugenfirer from comment #14)
> Hi Wyu,
> 
> 
> Thanks!
> 
> Can you please try and reproduce this bug with win-1.7.5-0.el6.noarch and
> latest virtio-win-prewhql build?
> 
> Thanks,
> Yan.

Yes, I will add comment when I have tried.
Comment 16 Yu Wang 2016-03-23 06:20:13 EDT
Hi Yan

I tried three times as comment#11 and comment#14 , but I still cannot reproduce this issue.

Thanks
wyu
Comment 17 Yan Vugenfirer 2016-03-23 07:24:46 EDT
Thanks!

I suggest to close the bug for now. Could be an issue with intermediate version that was used for upgrade.

Note You need to log in before you can comment on or make changes to this bug.