Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 873192

Summary: [whql][Netkvm] Job "NDISTest6.5-InvalidPackets" failed with WLK on win2k8/win2k8-R2
Product: Red Hat Enterprise Linux 6 Reporter: dawu
Component: virtio-winAssignee: Dmitry Fleytman <dfleytma>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4CC: acathrow, bcao, bsarathy, dfleytma, dyasny, juzhang, lijin, michen, rhod, yvugenfi
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-17 08:32:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
InvalidPacketsFail
none
InvalidPackets.htm
none
system-event-log none

Description dawu 2012-11-05 10:18:34 UTC
Description of problem:
Job of "Ethernet - NDISTest6.5" failed with WLK on win2k8/win2k8-R2

Version-Release number of selected component (if applicable):
kernel-2.6.32-338.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.331.el6.x86_64
virtio-win-prewhql-0.1-43

How reproducible:
3/3

Steps to Reproduce:
1. Start guest with CLI:
   /usr/libexec/qemu-kvm -M rhel6.4.0 -m 6G -smp 4 -cpu cpu64-rhel6,+x2apic -usbdevice tablet -drive file=win2k8-32-nic1.raw,if=none,id=drive-ide0-0-0,werror=stop,rerror=stop,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,sndbuf=0,id=hostnet0,vhost=on,script=/etc/qemu-ifup-private,downscript=no -device virtio-net-pci,netdev=hostnet0,mac=00:10:19:06:39:01,bus=pci.0,addr=0x4,id=virtio-net-pci0 -netdev tap,sndbuf=0,id=hostnet1,vhost=on,script=/etc/qemu-ifup-private,downscript=no -device virtio-net-pci,netdev=hostnet1,mac=00:10:19:06:39:02,bus=pci.0,addr=0x5,id=virtio-net-pci1 -netdev tap,sndbuf=0,id=hostnet2,script=/etc/qemu-ifup,downscript=no -device e1000,netdev=hostnet2,mac=00:10:19:16:39:03,bus=pci.0,addr=0x6 -uuid 01410ecf-365b-4629-83a6-c7fec786de81 -rtc base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection -chardev socket,id=111a,path=/tmp/monitor-win2k8-32-nic1,server,nowait -mon chardev=111a,mode=readline -spice port=5931,disable-ticketing -vga qxl -monitor stdio -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0

2.Run job of "Ethernet - NDISTest6.5"
3.Run sub job "Start NDISTest Client"
  
Actual results:
Job failed without any DTM logs, but only result error info as following:
-------------------------------------------------------------------------------
TaskResult Errors 
# Message Time Source Machine Name Root Cause 
1 11/5/2012 5:18:41 PM Execution Agent 2K8-32-43-NIC1 The Execute Task with Commandline

cmd /c ndistest.exe /logo /auto /client /target:Miniport /tc:{2DD2A524-E182-47C1-8FA8-2073FF990190} /TestEnvInfo:TestEnvInfo.xml /support:{9CB0A667-F509-4824-9220-9DE440E3447F},{89920EA9-811C-4588-8EE5-C0F11AF7DE43} /msg:{84EDCFFC-EE59-46A3-98C3-AAF567B907C1} /jobgroup:WHQL\Lan /ParaVirtualizationDriver:True 

Failed with ExitCode 1

 
Resolution 
The task exited with an ExitCode other than the ExpectedTaskExitCode. This may cause the Task to Fail if it is set to Fail On Exit Code
-------------------------------------------------------------------------------

Expected results:
Job should passed without any error.

Additional info:
No this issue on win2k3-32/64

Comment 2 dawu 2012-11-06 02:56:02 UTC
Created attachment 639059 [details]
cpk-win2k8-32-network-43.cpk

Please refer to the attached for the cpk file.

Best Regards,
Dawn

Comment 3 Yvugenfi@redhat.com 2012-11-14 16:59:44 UTC
We cannot find the exact cause of failure from the logs...

Do you see that the drivers are correctly installed on the VM?

Best regards,
Yan.

Comment 4 Yvugenfi@redhat.com 2012-11-15 10:05:50 UTC
In any case - please retest with: http://download.devel.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/44/win/

(this is also a fix for https://bugzilla.redhat.com/show_bug.cgi?id=873128 )

Comment 5 dawu 2012-11-15 10:27:52 UTC
(In reply to comment #3)
> We cannot find the exact cause of failure from the logs...
> 
> Do you see that the drivers are correctly installed on the VM?
Yes, the drivers are correctly installed on the VM, I retest again today with keeping eyes on the guests,find when the NDISTest Client is doing the last job of "postconfig", the NDISTest Server side finish "postconfig" job before Client and quickly quit.But the Client is still running "postconfig". 

And another thing is job "InvalidPackets.htm" fail for NDIS6.5, please refer to the attachment "InvalidPacketsFail.png"

Any way, I'll retest it on new driver 44, and update the result.

Thanks
Best Regards,
Dawn
> 
> Best regards,
> Yan.

Comment 6 dawu 2012-11-15 10:30:30 UTC
Created attachment 645562 [details]
InvalidPacketsFail

Comment 7 Yvugenfi@redhat.com 2012-11-26 17:41:31 UTC
Is it possible to run same test with REHL 6.3 host?

Thanks,
Yan.

Comment 8 Mike Cao 2012-11-27 01:56:06 UTC
(In reply to comment #7)
> Is it possible to run same test with REHL 6.3 host?
> 
> Thanks,
> Yan.

ongoing now

Comment 9 Yvugenfi@redhat.com 2012-11-27 13:30:18 UTC
(In reply to comment #6)
> Created attachment 645562 [details]
> InvalidPacketsFail

Can you double click on the test in the UI list and it should open the test log. Please save it and attach to BZ.

Thanks,
Yan.

Comment 10 dawu 2012-11-28 06:02:08 UTC
(In reply to comment #9)
> (In reply to comment #6)
> > Created attachment 645562 [details]
> > InvalidPacketsFail
> 
> Can you double click on the test in the UI list and it should open the test
> log. Please save it and attach to BZ.
> 
> Thanks,
> Yan.

(In reply to comment #9)
> (In reply to comment #6)
> > Created attachment 645562 [details]
> > InvalidPacketsFail
> 
> Can you double click on the test in the UI list and it should open the test
> log. Please save it and attach to BZ.
> 
> Thanks,
> Yan.

Hi Yan,

Please refer to the attachment of "InvalidPackets.htm" for details.

Best Regards,
Dawn

Comment 11 dawu 2012-11-28 06:04:05 UTC
Created attachment 653272 [details]
InvalidPackets.htm

Comment 12 Yvugenfi@redhat.com 2012-11-28 14:19:18 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #6)
> > > Created attachment 645562 [details]
> > > InvalidPacketsFail
> > 
> > Can you double click on the test in the UI list and it should open the test
> > log. Please save it and attach to BZ.
> > 
> > Thanks,
> > Yan.
> 
> (In reply to comment #9)
> > (In reply to comment #6)
> > > Created attachment 645562 [details]
> > > InvalidPacketsFail
> > 
> > Can you double click on the test in the UI list and it should open the test
> > log. Please save it and attach to BZ.
> > 
> > Thanks,
> > Yan.
> 
> Hi Yan,
> 
> Please refer to the attachment of "InvalidPackets.htm" for details.
> 
> Best Regards,
> Dawn

Thanks!

Any news on using RHEL6.3 host?

Comment 13 Yvugenfi@redhat.com 2012-11-28 15:38:52 UTC
Bridge filtering settings:

Good settings are:
# sysctl -a|grep bridge
net.bridge.bridge-nf-call-arptables = 0 
net.bridge.bridge-nf-call-iptables = 0 
net.bridge.bridge-nf-call-ip6tables = 0

Comment 14 dawu 2012-11-29 07:58:48 UTC
Hi Yan,

Tried to run the same test with rhel6.3 host, still hit the same issue with rhel6.4 host.

Best Regards,
Dawn

Comment 17 Yvugenfi@redhat.com 2012-12-03 11:27:31 UTC
(In reply to comment #5)
> (In reply to comment #3)
> > We cannot find the exact cause of failure from the logs...
> > 
> > Do you see that the drivers are correctly installed on the VM?
> Yes, the drivers are correctly installed on the VM, I retest again today
> with keeping eyes on the guests,find when the NDISTest Client is doing the
> last job of "postconfig", the NDISTest Server side finish "postconfig" job
> before Client and quickly quit.But the Client is still running "postconfig". 
> 
> And another thing is job "InvalidPackets.htm" fail for NDIS6.5, please refer
> to the attachment "InvalidPacketsFail.png"
> 

I don't think it is a problem. Postconfig is a local task of removing the protocol driver and reconfiguring the VM if the test did something with it. It is expected for the server to quit. Also the logs are collected from the client side only.


> Any way, I'll retest it on new driver 44, and update the result.
> 
> Thanks
> Best Regards,
> Dawn
> > 
> > Best regards,
> > Yan.

Comment 18 Yvugenfi@redhat.com 2012-12-03 15:34:02 UTC
Please check the event log for traces of the system crash.

Thanks,
Yan.

Comment 19 dawu 2012-12-04 09:02:36 UTC
(In reply to comment #18)
> Please check the event log for traces of the system crash.
> 
> Thanks,
> Yan.

Hi yan,

Please refer to the attached "system-event-log.evtx" for logs.

Best Regards,
Dawn

Comment 20 dawu 2012-12-04 09:04:03 UTC
Created attachment 657326 [details]
system-event-log

Comment 21 Yvugenfi@redhat.com 2012-12-04 12:08:05 UTC
Hi Dawn,

I see in the event log following errors: 
"The server could not bind to the transport \Device\NetBT_Tcpip_{003E256F-3421-468A-BB7B-AC6F5EFAE7DA} because another computer on the network has the same name.  The server could not start." (computer name "2k8-32-1-46.virtio.com").


Considering that the DTM uses computer names - this might cause some issues. Can you double check that the VMs used for test and possibly others VMs in the network don't have same name.

Thanks,
Yan.

Comment 22 Yvugenfi@redhat.com 2012-12-04 12:08:29 UTC
Hi Dawn,

I see in the event log following errors: 
"The server could not bind to the transport \Device\NetBT_Tcpip_{003E256F-3421-468A-BB7B-AC6F5EFAE7DA} because another computer on the network has the same name.  The server could not start." (computer name "2k8-32-1-46.virtio.com").


Considering that the DTM uses computer names - this might cause some issues. Can you double check that the VMs used for test and possibly others VMs in the network don't have same name.

Thanks,
Yan.

Comment 23 dawu 2012-12-05 02:46:35 UTC
(In reply to comment #22)
> Hi Dawn,
> 
> I see in the event log following errors: 
> "The server could not bind to the transport
> \Device\NetBT_Tcpip_{003E256F-3421-468A-BB7B-AC6F5EFAE7DA} because another
> computer on the network has the same name.  The server could not start."
> (computer name "2k8-32-1-46.virtio.com").
> 
> 
> Considering that the DTM uses computer names - this might cause some issues.
> Can you double check that the VMs used for test and possibly others VMs in
> the network don't have same name.
> 
> Thanks,
> Yan.

Hi Yan, 

Yes, actually I saw the logs yesterday, and felt strange since we named the guest as "system name + driver name + build Number + NIC1/2", so shouldn't have the duplicate name,and in the domain if the name duplicate, system will prompt you computer name duplicate. So I still checked two times, didn't find the same name exist for all guest (win2k8-32/64 win2k8-R2).

Best Regards,
Dawn

Comment 24 Dmitry Fleytman 2012-12-06 12:50:30 UTC
Since this could be a setup problem (transport binding failure), please try the following:
    1. Close all ndis-test related applications running on HCK clients
    2. Uninstall NetKVM driver for all RedHat adapters on all clients(Device Manager --> Rings Click --> Uninstall --> Delete driver software)
    3. Reboot all clients
    4. Install drivers again for all Redhat adapters on all clients
    5. Reboot all clients
    6. Try to repeat this test

We saw a few similar problems, this sequence usually helps.

Comment 25 Mike Cao 2012-12-06 14:34:03 UTC
(In reply to comment #24)
> Since this could be a setup problem (transport binding failure), please try
> the following:
>     1. Close all ndis-test related applications running on HCK clients

We are asked to Run netkvm jobs on WLK ,Could you confirm whether the following steps works on WLK as well ?

Mike
>     2. Uninstall NetKVM driver for all RedHat adapters on all clients(Device
> Manager --> Rings Click --> Uninstall --> Delete driver software)
>     3. Reboot all clients
>     4. Install drivers again for all Redhat adapters on all clients
>     5. Reboot all clients
>     6. Try to repeat this test
> 
> We saw a few similar problems, this sequence usually helps.

Comment 26 Dmitry Fleytman 2012-12-09 15:45:28 UTC
(In reply to comment #25)
> (In reply to comment #24)
> > Since this could be a setup problem (transport binding failure), please try
> > the following:
> >     1. Close all ndis-test related applications running on HCK clients
> 
> We are asked to Run netkvm jobs on WLK ,Could you confirm whether the
> following steps works on WLK as well ?
> 

Yes, these steps should work on WLK as well.

> Mike
> >     2. Uninstall NetKVM driver for all RedHat adapters on all clients(Device
> > Manager --> Rings Click --> Uninstall --> Delete driver software)
> >     3. Reboot all clients
> >     4. Install drivers again for all Redhat adapters on all clients
> >     5. Reboot all clients
> >     6. Try to repeat this test
> > 
> > We saw a few similar problems, this sequence usually helps.

Comment 27 Yvugenfi@redhat.com 2012-12-11 09:17:02 UTC
Please try the steps in comment #24

Comment 28 dawu 2012-12-11 09:47:12 UTC
(In reply to comment #27)
> Please try the steps in comment #24

Hi Yan,

Tried with the steps in comment #24 to re-install the netkvm driver, still hit the same issue.

Best Regards,
Dawn

Comment 33 Mike Cao 2012-12-12 05:46:34 UTC
Looks this bug should be filter by https://sysdev.microsoft.com/en-US/Hardware/EC/ECDetails.aspx?id=2966 

I will research why filter does not help.

Comment 34 Mike Cao 2012-12-12 08:05:56 UTC
Retest it on Build49 on win2k8R2 w/ the driver located in win7
I don't think "InvalidPackages" is the key point cause job failed ,it can be filter by https://sysdev.microsoft.com/en-US/Hardware/EC/ECDetails.aspx?id=2966  from my test 


The job failed due to "Make Sure communication helpers stats and adapter stats match" sub job 

Following is the logs :

Failed 
Start Test 12/12/2012 1:06:57.656 PM Make sure communication helper stats and adapter stats match. 
Error 12/12/2012 1:06:57.656 PM Adapter statistics and comm helper statistics do not match! 
File:    Line: 0 
Error Type:   WIN32 
Error Code:   0x50012 
Error Text:   There are no more files. 
End Test 12/12/2012 1:06:57.656 PM Make sure communication helper stats and adapter stats match. 
Result:   Fail 
Repro:   ndistest.exe /logo /auto /client /target:Miniport /tc:{653340A5-E5F0-424E-B98D-C59169DFD541} /TestEnvInfo:TestEnvInfo.xml /support:{F64CE5E3-56C5-4810-BB46-225AC67D6F3E},{653340A5-E5F0-424E-B98D-C59169DFD541} /msg:{ACC8DD1F-8734-4CA9-8352-44218376E597} /jobgroup:WHQL\Lan /ParaVirtualizationDriver:True

Comment 36 Mike Cao 2012-12-14 07:42:31 UTC
(In reply to comment #34)
> Retest it on Build49 on win2k8R2 w/ the driver located in win7
> I don't think "InvalidPackages" is the key point cause job failed ,it can be
> filter by
> https://sysdev.microsoft.com/en-US/Hardware/EC/ECDetails.aspx?id=2966  from
> my test 
> 
> 
> The job failed due to "Make Sure communication helpers stats and adapter
> stats match" sub job 
> 
> Following is the logs :
> 
> Failed 
> Start Test 12/12/2012 1:06:57.656 PM Make sure communication helper stats
> and adapter stats match. 
> Error 12/12/2012 1:06:57.656 PM Adapter statistics and comm helper
> statistics do not match! 
> File:    Line: 0 
> Error Type:   WIN32 
> Error Code:   0x50012 
> Error Text:   There are no more files. 
> End Test 12/12/2012 1:06:57.656 PM Make sure communication helper stats and
> adapter stats match. 
> Result:   Fail 
> Repro:   ndistest.exe /logo /auto /client /target:Miniport
> /tc:{653340A5-E5F0-424E-B98D-C59169DFD541} /TestEnvInfo:TestEnvInfo.xml
> /support:{F64CE5E3-56C5-4810-BB46-225AC67D6F3E},{653340A5-E5F0-424E-B98D-
> C59169DFD541} /msg:{ACC8DD1F-8734-4CA9-8352-44218376E597} /jobgroup:WHQL\Lan
> /ParaVirtualizationDriver:True

Re-run this job on 2k8R2 5 times ,one time passed .
win2k8-32 passed as well ,I will run this job on win2k8-64 Once the win2k8R2 netkvm whql test finished .

Best Regards,
Mike

Comment 38 Mike Cao 2012-12-17 08:32:39 UTC
(In reply to comment #36)

> Re-run this job on 2k8R2 5 times ,one time passed .
> win2k8-32 passed as well ,I will run this job on win2k8-64 Once the win2k8R2
> netkvm whql test finished .
> 
> Best Regards,
> Mike

Did not hit this issue on build49 over win7 32/64 ,win2k8 32/64/R2 bit.
There must be a configuration issue on QE AD testing environment ,I will re-setup it after all block&netkvm job submit to MSFT.

Closing this one

Comment 39 Dmitry Fleytman 2012-12-20 12:32:59 UTC
Also this erratum will probably help: https://sysdev.microsoft.com/en-US/Hardware/EC/ECDetails.aspx?id=2966