Bug 920472

Created attachment 709273 [details]
2012log

This issue also happened on other platforms and other jobs in OVS configuration environment, it always caused by the error in logs: "xx total breakpoints were hit in the protocol driver while this test was executing", following is the details for other platforms and jobs:

win2k8-32: NDISTest 6.5 - [2 Machine] - SingleEtherType 
           NDISTest 6.5 - [2 Machine] - Stats

win2k8-64: NDISTest 6.5 - [2 Machine] - Stats

win8-32:   NDISTest 6.5 - [2 Machine] - Stats
           NDISTest 6.5 - [2 Machine] - PacketFilters
           NDISTest 6.5 - [2 Machine] - MultiCastAddress
win8-64:   NDISTest 6.5 - [2 Machine] - MultiCastAddress

above jobs were run many times (5~6),never passed,but some other NDISTest 6.5 jobs also hit the same issue, but when ran 2~3 times or even 5 times, it passed finally.

Thanks
Best Regards,
Dawn

Hello,

This test passes on our setup with NetKVM build65 and OVS openvswitch-1.7.1-7.el6.
Please, retest.

Dmitry

in ovs environment,many jobs failed in build 65 netkvm HCK test,the verison of ovs is openvswitch-1.9.0-3.el6.x86_64.
many jobs have error in logs: "xx total breakpoints were hit in the protocol driver while this test was executing", following is the related platforms and jobs:
win2012:
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - PacketFilters
NDISTest 6.5 - [2 Machine] - GlitchFreeDevice

win2k8-32: 
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - Stats

win2k8-64: 
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - PacketFilters
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - MultiCastAddress

w2k8-R2:
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - reset
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - MultiCastAddress
NDISTest 6.5 - [2 Machine] - GlitchFreeDevice


win7-32:
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - LinkCheck
NDISTest 6.5 - [2 Machine] - InvalidPackets
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - GlitchFreeDevice
NDISTest 6.5 - [2 Machine] - MultiCastAddress

win7-64:
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - InvalidPackets

win8-32:
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - InvalidPackets

win8-64:
NDISTest 6.5 - [2 Machine] - Stats
NDISTest 6.5 - [2 Machine] - SingleEtherType 
NDISTest 6.5 - [2 Machine] - InvalidPackets
NDISTest 6.5 - [2 Machine] - GlitchFreeDevice
NDISTest 6.5 - [2 Machine] - MultiCastAddress

Hi,

Thanks for the report. Please attach HCK logs (.hckx) for failed tests.
Also, could you please run the same tests on the same VMs/Hosts with Linux bridge and with openvswitch-1.7.1-7.el6?

Thanks in advance,
Dmitry

Created attachment 770223 [details]
win2012-ovs-netkvm-build65.hckx

attacment "win2012-ovs-netkvm-build65.hckx" is the HCK logs for win2012,I will uplaod more if it is needed.

(In reply to Dmitry Fleytman from comment #7)
> Hi,
> 
> Thanks for the report. Please attach HCK logs (.hckx) for failed tests.
> Also, could you please run the same tests on the same VMs/Hosts with Linux
> bridge and with openvswitch-1.7.1-7.el6?
> 
> Thanks in advance,
> Dmitry

Run the same tests on win2012 guest:
1.With Linux bridge,not hit this issue;
2.With openvswitch-1.7.1-7.el6,only one job "NDISTest 6.5 - [2 Machine] - GlitchFreeDevice" passed,others are still failed with the same error "xx total breakpoints were hit in the protocol driver while this test was executing";
And please note that with openvswitch-1.9.0-3.el6.x86_64,there are also a few jobs can pass if we run several times.

The root cause of this (and some other) test failures on OVS configurations is out-of-order packets. Indeed, there is a problem in openvswitch - its user mode service may reorder incoming packets on some systems.

I've opened discussion on OVS mailing list and submitted a patch that fixes a user mode service probem:

http://openvswitch.org/pipermail/dev/2013-July/029742.html
http://openvswitch.org/pipermail/dev/2013-July/029743.html

While this patch fixes part of problems there are other scenarios that need to be investigated, not changing bug state for now.

This bug was fixed in openvswitch upstream.

commit 04a19fb8f4b8ba19a9805906aac7b30b65b57206
Author: Ben Pfaff <blp>
Date:   Thu Sep 19 11:03:47 2013 -0700

    ofproto-dpif-upcall: Forward packets in order of arrival.

    Until now, the code in ofproto-dpif-upcall (and the code that preceded it
    in ofproto-dpif) obtained a batch of incoming packets, inserted them into
    a hash table based on hashes of their flows, processed them, and then
    forwarded them in hash order.  Usually this maintains order within a single
    network connection, but because OVS's notion of a flow is so fine-grained,
    it can reorder packets within (e.g.) a TCP connection if two packets
    handled in a single batch have (e.g.) different ECN values.

    This commit fixes the problem by making ofproto-dpif-upcall always forward
    packets in the same order they were received.

    This is far from the minimal change necessary to avoid reordering packets.
    I think that the code is easier to understand afterward.

    Reported-by: Dmitry Fleytman <dfleytma>
    Signed-off-by: Ben Pfaff <blp>
    Acked-by: Jarno Rajahalme <jrajahalme>

 I'm afraid that the bug wasn't fix the issue extremely,and QE will upload the related hck files to the bug as well.The issue was still there via builds openvswitch-2.0.0-7.el7.x86_64
kernel-3.10.0-84.el7.x86_64 or kernel-3.10.0-105.el7.x86_64
qemu-kvm-rhev-1.5.3-46.el7.x86_64

win8-32     - Stats
              SingleEtherType
              InvalidPackets
Win2k8-R2   - SingleEtherType

win7-64     - InvalidPackets
              SingleEtherType

win2012-R2  - Stats
              SingleEtherType

Created attachment 874305 [details]
2012R2

Created attachment 874309 [details]
764

Created attachment 874310 [details]
2k8R2

Re-test the bug with openvswitch 
openvswitch-2.4.0-1.el7.x86_64.rpm
kernel-3.10.0-330.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7.x86_64
The job named stats still failed and upload log to the bug.