Bug 1780337 - There should be a networking stress test harness that helps catch networking flakes in a more controlled environment
Summary: There should be a networking stress test harness that helps catch networking ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.3.z
Assignee: Clayton Coleman
QA Contact: Mike Fiedler
URL:
Whiteboard: SDN-CI-IMPACT
Depends On: 1780334
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-05 17:32 UTC by Clayton Coleman
Modified: 2020-07-07 13:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1780334
Environment:
Last Closed: 2020-07-07 13:43:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24260 0 'None' closed Bug 1780337: Add a network stress test and --count to repeat tests 2020-07-07 13:42:40 UTC

Description Clayton Coleman 2019-12-05 17:32:31 UTC
+++ This bug was initially created as a clone of Bug #1780334 +++

As we have begun rolling out OVN (and over the lifetime of openshift-sdn) we often have subtle networking bugs that only materialize in other test suites in a general fashion that are difficult to root cause. This hampers our detect-triage-fix loop, and also causes a high amount of team to team overhead.  This also forces the networking team into a lot of high overhead interactions that reduces their time spent on actually triaging and fixing their issues (the "it's always networking" mindset).

To better isolate specific testing, we should create a network stress test harness that can evolve to provide more specific detection and resolution of networking issues as we go.  This can start simply - a test that runs the networking e2es repeatedly in parallel - and grow over time to offer more sophisticated invariant checking (a long running test that verifies that all pods can reach all targets).

We should add the test harness as an e2e suite and add a release periodic and PR job for openshift/sdn and ovn-kube that allows us to trigger the test.  We will have follow up work that ensures the tests are more effective, and probably want to look at process changes that ensure when we hit flakes that weren't caught by the stress test we add a new case. 

Things we can do later:

1. run an invariant checker (as a test, or part of the monitor, or as a disruptive test) that verifies the environment is currently working (exec into a pod and make sure it can curl masters over network, known working pods over service, directly reach pods over pod network, or access the host they are running on via host network)
2. add new types of e2e tests that try disruptive style things (delete the ovn-kube pod on a node and verify no failures are detected)
3. better instrument upgrades to verify that no connections are dropped, like we do for service load balancers
4. ... ?

--- Additional comment from Clayton Coleman on 2019-12-05 12:29:45 EST ---

For right now getting an openshift-sdn and ovn-kube stress test release job is the primary goal, then we will assess further.

Comment 1 Mike Fiedler 2020-01-15 14:18:02 UTC
The PR linked in this bz which is in POST is for the release-4.3 branch.   Setting target release to match.

Comment 2 Ben Bennett 2020-05-08 22:55:20 UTC
@clayton - Did you want this backport to go in?  If so, I'll take the bug.

Comment 3 Clayton Coleman 2020-07-07 13:43:17 UTC
I don't think this needs more backport.  We're good.


Note You need to log in before you can comment on or make changes to this bug.