Bug 684082
Summary: | [WHQL] NDIS Test 6.0 failed for win7 and win2k8 | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Qunfang Zhang <qzhang> | ||||||
Component: | virtio-win | Assignee: | Yvugenfi <yvugenfi> | ||||||
Status: | CLOSED NOTABUG | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
Severity: | medium | Docs Contact: | |||||||
Priority: | medium | ||||||||
Version: | 6.1 | CC: | afrenkel, dawu, juzhang, rhod, tburke, vrozenfe | ||||||
Target Milestone: | rc | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2011-04-03 08:45:34 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 580954 | ||||||||
Attachments: |
|
Description
Qunfang Zhang
2011-03-11 03:57:39 UTC
DTM Log for win7-32: Start Test 3/10/2011 5:07:13.579 PM 2c_mini6stress Message 3/10/2011 5:07:13.579 PM This tests the stress send/receive code paths of NDIS 6.0 miniport drivers. End Test 3/10/2011 5:09:49.579 PM 2c_mini6stress Result: Fail Start Test 3/10/2011 5:07:29.579 PM Single open stress test 1 Error 3/10/2011 5:07:35.579 PM Not a single NET_BUFFER_LIST/NET_BUFFER sent was received File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 End Test 3/10/2011 5:07:35.579 PM Single open stress test 1 Result: Fail Repro: 2c_mini6stress Start Test 3/10/2011 5:08:59.579 PM Random unidirectional Test Error 3/10/2011 5:09:21.579 PM Not a single NET_BUFFER_LIST/NET_BUFFER sent was received File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 End Test 3/10/2011 5:09:21.579 PM Random unidirectional Test Result: Fail Repro: 2c_mini6stress Start Test 3/10/2011 5:09:21.579 PM Random bidirectional Test Error 3/10/2011 5:09:45.579 PM Not a single NET_BUFFER_LIST/NET_BUFFER sent was received File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 End Test 3/10/2011 5:09:49.579 PM Random bidirectional Test Result: Fail Repro: 2c_mini6stress Start Test 3/10/2011 5:36:34.579 PM 2c_mini6sendrecvreply Message 3/10/2011 5:36:34.579 PM This is a bidirectional send receive test for Ndis 6.0 miniports. It sends packets from the test adapter to the support adapter, which the support adapter is expected to reply to. At the same time, it sends packets from the support adapter to the test adapter, which the test adapter is expected to reply to. This should result in continuous bidirectional traffic between the two adapters. End Test 3/10/2011 5:36:59.579 PM 2c_mini6sendrecvreply Result: Fail Start Test 3/10/2011 5:36:48.579 PM Starting Test->Supp->Test Error 3/10/2011 5:36:54.579 PM Support card did not receive any replies sent from the Test card File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 Error 3/10/2011 5:36:56.579 PM Support card did not receive any packets sent from the Test card File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 Error 3/10/2011 5:36:56.579 PM Support card did not send any reply net buffer lists File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 Error 3/10/2011 5:36:56.579 PM Test card did not receive any replies sent from the Support card File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 End Test 3/10/2011 5:36:56.579 PM Starting Test->Supp->Test Result: Fail Repro: 2c_mini6sendrecvreply Start Test 3/10/2011 6:03:25.579 PM 2c_mini6loadunload Message 3/10/2011 6:03:25.579 PM This script verifies the ability to detect load and unload the miniport driver. It loads/unloads all local test and support adapters (simultaneously) and in the end ensures that the test adapter is able to talk to either a remote (or if no remote adapter selected, a local support adapter). It logs a failure if any of the adapters fails to load, or if the communication test at the end of the test run fails. It spawns additional script jobs to perform the simultaneous loading/unloading of adapters. End Test 3/10/2011 6:08:42.579 PM 2c_mini6loadunload Result: Fail Start Test 3/10/2011 6:08:36.579 PM Sending Packets Error 3/10/2011 6:08:39.579 PM Test <-> Support card communication failed after load/unload File: Line: 0 Error Type: NT_STATUS Error Code: 0x15b38 Error Text: Error 0x00015b38 End Test 3/10/2011 6:08:39.579 PM Sending Packets Result: Fail Repro: 2c_mini6loadunload (In reply to comment #0) > Description of problem: > Ethernet NDIS Test 6.0 failed for win7 and win2k8-64 guest. For win2k3 and xp > guests, will update comment after get results. > For win7-32 guest, after the job "Start NDIS Test Client" failed and run > "disable driver verifier", guest always hang for a while during reboot. Then > after it boot up, it can not auto-login with the DTMLLUAdminUser account. > > DTM log will be attached soon. > > Version-Release number of selected component (if applicable): > kernel-2.6.32-118.el6.x86_64 > qemu-kvm-0.12.1.2-2.148.el6.x86_64 > virtio-win-1.1.20 > > How reproducible: > win7-32: 2/2 > win7-64: 2/2 > win2k8-64: 2/2 > win2k8-32: pass > win2k8-R2: pass > > Steps to Reproduce: > 1. Prepare for whql nic test environemnt. > 2. Run Ethernet NDIS Test 6.0. > 3. > > Actual results: > The job always failed. > > Expected results: > Job can pass without any error. > > Additional info: This issue does not reproduce on win2k3/winxp, job passed without error. First off all I see several fails in your description. But the main thing that should be checked as part of the setup is related to: "PM Not a single NET_BUFFER_LIST/NET_BUFFER sent was received" Please check that NDIS client and server can receive pings from each other through the following: from e1000 to e1000 and from each one of the virtio adapters to another (you can disable unused adapters during this check). Created attachment 485650 [details]
NetworkFailToEnableWithIPV4disabled
(In reply to comment #4) > First off all I see several fails in your description. But the main thing that > should be checked as part of the setup is related to: > "PM Not a single NET_BUFFER_LIST/NET_BUFFER sent was > received" > > > Please check that NDIS client and server can receive pings from each other > through the following: from e1000 to e1000 and from each one of the virtio > adapters to another (you can disable unused adapters during this check). Hi Yan, 1. I checked the NDIS client,this issue caused by the test card failed to be enabled successfully when doing network disable/enable work repeatedly during running NDIS6.0, and the ipv4/ipv6 was disabled at this time,so failed to get ip address,please refer to the attachment of "NetworkFailToEnableWithIPV4disabled.png". 2. BSOD happened when doing reboot job in NDIS6.0 testing if test card failed to be enabled, I'm not sure whether this BSOD issue is related with problem of card enable failure.Please refer to the attachment of "BSOD-enalbeFailed.png",for dump file please get it from http://10.66.65.33/dump-file/DumpFile_684082.tar.gz Best Regard, Dawn Created attachment 485651 [details]
BSOD-enalbeFailed
(In reply to comment #6) > (In reply to comment #4) > > First off all I see several fails in your description. But the main thing that > > should be checked as part of the setup is related to: > > "PM Not a single NET_BUFFER_LIST/NET_BUFFER sent was > > received" > > > > > > Please check that NDIS client and server can receive pings from each other > > through the following: from e1000 to e1000 and from each one of the virtio > > adapters to another (you can disable unused adapters during this check). > > Hi Yan, > > 1. I checked the NDIS client,this issue caused by the test card failed to be > enabled successfully when doing network disable/enable work repeatedly during > running NDIS6.0, and the ipv4/ipv6 was disabled at this time,so failed to get > ip address,please refer to the attachment of > "NetworkFailToEnableWithIPV4disabled.png". This issue happened in sub job "Start NDISTest Client" with configuration file of "1c_ndisrequestcov.html". The whole job can pass with manually selecting ipv4/ipv6 checkbox and enable network again in timely when network fails to be enabled at this step. Best Regards, Dawn > > 2. BSOD happened when doing reboot job in NDIS6.0 testing if test card failed > to be enabled, I'm not sure whether this BSOD issue is related with problem of > card enable failure.Please refer to the attachment of > "BSOD-enalbeFailed.png",for dump file please get it from > http://10.66.65.33/dump-file/DumpFile_684082.tar.gz > > Best Regard, > Dawn It seems that the blue screen is not related to network, but rather to VGA driver. I will continue to investigate the failures. Crash dump analysis: 0: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* THREAD_STUCK_IN_DEVICE_DRIVER (ea) The device driver is spinning in an infinite loop, most likely waiting for hardware to become idle. This usually indicates problem with the hardware itself or with the device driver programming the hardware incorrectly. If the kernel debugger is connected and running when watchdog detects a timeout condition then DbgBreakPoint() will be called instead of KeBugCheckEx() and detailed message including bugcheck arguments will be printed to the debugger. This way we can identify an offending thread, set breakpoints in it, and hit go to return to the spinning code to debug it further. Because KeBugCheckEx() is not called the .bugcheck directive will not return bugcheck information in this case. The arguments are already printed out to the kernel debugger. You can also retrieve them from a global variable via "dd watchdog!g_WdBugCheckData l5" (use dq on NT64). On MP machines (OS builds <= 3790) it is possible to hit a timeout when the spinning thread is interrupted by hardware interrupt and ISR or DPC routine is running at the time of the bugcheck (this is because the timeout's work item can be delivered and handled on the second CPU and the same time). If this is the case you will have to look deeper at the offending thread's stack (e.g. using dds) to determine spinning code which caused the timeout to occur. Arguments: Arg1: fffffa800348ba80, Pointer to a stuck thread object. Do .thread then kb on it to find the hung location. Arg2: fffffa80034c7990, Pointer to a DEFERRED_WATCHDOG object. Arg3: fffffa80034bf850, Pointer to offending driver name. Arg4: 0000000000000001, Number of times this error occurred. If a debugger is attached, this error is not always fatal -- see DESCRIPTION below. On the blue screen, this will always equal 1. Debugging Details: ------------------ Page 2170e not present in the dump file. Type ".hh dbgerr004" for details Page 23162 not present in the dump file. Type ".hh dbgerr004" for details ERROR - could not read driver name for bugcheck parameter 3 Page 23162 not present in the dump file. Type ".hh dbgerr004" for details Page 23162 not present in the dump file. Type ".hh dbgerr004" for details PEB is paged out (Peb.Ldr = 000007ff`fffdf018). Type ".hh dbgerr001" for details PEB is paged out (Peb.Ldr = 000007ff`fffdf018). Type ".hh dbgerr001" for details FAULTING_THREAD: fffffa800348ba80 FAULTING_IP: framebuf!bInitSURF+47 fffff960`009d22e3 3bc5 cmp eax,ebp IMAGE_NAME: framebuf.dll DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bc587 MODULE_NAME: framebuf FAULTING_MODULE: fffff960009d0000 framebuf DEFAULT_BUCKET_ID: GRAPHICS_DRIVER_FAULT BUGCHECK_STR: 0xEA PROCESS_NAME: csrss.exe CURRENT_IRQL: 1 LAST_CONTROL_TRANSFER: from fffff88001995c54 to fffff800026cd740 STACK_TEXT: fffff880`040684f8 fffff880`01995c54 : 00000000`000000ea fffffa80`0348ba80 fffffa80`034c7990 fffffa80`034bf850 : nt!KeBugCheckEx fffff880`04068500 fffff800`026aa0b7 : fffffa80`0348ba80 fffffa80`0348bad0 00000000`00000001 fffff800`02615939 : VIDEOPRT!WdpKernelApc+0x2e8 fffff880`04068a30 fffff800`026aa477 : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDeliverApc+0x1d7 fffff880`04068ab0 fffff800`0262b9d6 : 00000000`00000010 00000000`00000001 fffff800`02614000 fffff800`0262de3e : nt!KiApcInterrupt+0xd7 fffff880`04068c40 fffff800`0262c03f : fffff800`0263bf60 00000000`00000001 00000000`00000000 fffff800`02614000 : hal!x86BiosWriteIoSpace+0x15e fffff880`04068c90 fffff800`0262be30 : fffff800`0263bf60 fffff880`04068d60 00000000`00000000 fffff880`04068d60 : hal!XmEmulateStream+0x10f fffff880`04068cc0 fffff800`026251c5 : 00000000`00000000 00000000`00004115 fffffa80`034c8fc0 fffff880`04068db0 : hal!x86BiosExecuteInterruptShadowed+0xfc fffff880`04068cf0 fffff880`0199f0a7 : 00000000`00000000 00000000`00000000 fffffa80`0278c300 00000000`00000000 : hal!x86BiosCall+0x45 fffff880`04068d30 fffff880`03c6830d : 00000000`00000007 fffffa80`029fe618 00000000`00000000 fffff800`0285c880 : VIDEOPRT!VideoPortInt10+0x9f fffff880`04068d90 fffff880`03c68d81 : fffffa80`0278c380 00000000`00000000 00000000`00000000 fffff880`00e1d9ac : vgapnp!VgaSetMode+0x115 fffff880`04068df0 fffff880`019aaeb3 : fffffa80`0278c500 fffff880`00000000 00000000`00000000 00000000`00000000 : vgapnp!VgaStartIO+0x2e5 fffff880`04068e90 fffff960`00084abe : 00000000`00000004 fffffa80`0278c350 fffff900`c0083de8 fffffa80`029fe618 : VIDEOPRT!pVideoPortDispatch+0x3af fffff880`04068f40 fffff960`001f83a8 : fffff900`c0083db0 00000000`00000000 fffff900`c00d8010 00000000`00000001 : win32k!GreDeviceIoControlEx+0xd6 fffff880`04068fe0 fffff960`009d22e3 : fffff900`c00c4000 00000000`00000000 00000009`0000000c 00000000`00000000 : win32k!EngDeviceIoControl+0x3c fffff880`04069030 fffff960`009d1867 : 00000000`00000000 fffff900`c0083db0 fffff900`c0083db0 fffff960`0008ff9e : framebuf!bInitSURF+0x47 fffff880`040690b0 fffff960`00079cb6 : fffffa80`034b3d60 fffff900`c00d8920 fffff800`02847e80 fffff800`026d9de0 : framebuf!DrvEnableSurface+0x1f fffff880`04069100 fffff960`002e3de2 : fffffa80`034b3d60 fffff900`c00d8920 00000000`40210088 fffff900`64776447 : win32k!WatchdogDrvEnableSurface+0x36 fffff880`04069140 fffff960`000882f2 : fffff900`c00c4000 fffff880`040692b8 fffff900`c00c4000 fffff900`c00bdec0 : win32k!PanEnableSurface+0x5a fffff880`04069210 fffff960`00080fae : fffff900`c00c4000 00000000`00000001 00000000`00000000 fffffa80`029fe070 : win32k!PDEVOBJ::bMakeSurface+0x4a fffff880`04069240 fffff960`0007c1ea : fffff900`c00be420 fffff900`c00be420 00000000`00000001 00000000`00000000 : win32k!hCreateHDEV+0xb8e fffff880`040693a0 fffff960`00234b83 : 00000000`00000000 00000000`ffffffff 00000000`00000000 00000000`00000000 : win32k!DrvCreateMDEV+0x7d6 fffff880`04069640 fffff960`0007a9dd : fffff880`00000000 ffffffff`00000000 00000000`00000000 00000000`00000000 : win32k!DrvInternalChangeDisplaySettings+0x7a3 fffff880`04069880 fffff960`0007a200 : 00000000`00000128 00000000`00000000 00000000`00000000 00000000`00000000 : win32k!DrvChangeDisplaySettings+0x62d fffff880`04069a60 fffff960`000815da : 00000000`00000000 00000000`ffffaecc 00000000`00000000 00000000`00000000 : win32k!InitVideo+0x1f8 fffff880`04069b60 fffff960`00075841 : fffff900`c0600000 fffff8a0`0166ec60 fffffa80`0348ab30 00000000`0000004c : win32k!UserInitialize+0x28a fffff880`04069be0 fffff800`026cc993 : fffffa80`0348ba80 00000000`001f0003 00000000`00000007 fffff880`04069c01 : win32k!NtUserInitialize+0xc1 fffff880`04069c20 000007fe`fd7233da : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 00000000`001bf4f8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x7fe`fd7233da > The whole job can pass with manually selecting ipv4/ipv6 checkbox and enable
> network again in timely when network fails to be enabled at this step.
>
If this is the case ,than it means that the network card was left in incorrect state by the test suit and that caused the failure in the next tests.
Please insure that the network card configured correctly before starting the tests (either by enabling the protocol driver stack or by reinstalling the card and checking that it is functional).
Also please provide complete CPK file from the submission so the test that failed first could be captured.
Question from Arkady - could you try t re-run the tests on freshly installed VMs (DMT client + latest drivers). Yan run the test yesterday ( from his e-mail ) : Currently NDIS test is running on two win2008R2 VMs on boiler. Access using VNC: (client) boiler.eng.lab.tlv.redhat.com:2 (server) boiler.eng.lab.tlv.redhat.com:3 Password for administrator: Qum5net (with capital Q) Additional boiler info: http://cleo.tlv.redhat.com/qumrawiki/Boiler%20setup Best regards, Yan. There are no problems in it Arkady (In reply to comment #11) > Question from Arkady - could you try t re-run the tests on freshly installed > VMs (DMT client + latest drivers). Retested with latest build virtio-win-prewhql-0.1-8 on win2k8-R2 and win7-64,NDIS Test 6.0 passed without any error,this issue does not reproduce,I will continue test this job on the left platforms and update this issue later. Following is the test environment: virtio-win-prewhql-0.1-8 kernel-2.6.32-120.el6.x86_64 qemu-kvm-0.12.1.2-2.152.el6.x86_64 NDIS 6.0 passed on my setup as well. Closing the bug |