As we have begun rolling out OVN (and over the lifetime of openshift-sdn) we often have subtle networking bugs that only materialize in other test suites in a general fashion that are difficult to root cause. This hampers our detect-triage-fix loop, and also causes a high amount of team to team overhead. This also forces the networking team into a lot of high overhead interactions that reduces their time spent on actually triaging and fixing their issues (the "it's always networking" mindset). To better isolate specific testing, we should create a network stress test harness that can evolve to provide more specific detection and resolution of networking issues as we go. This can start simply - a test that runs the networking e2es repeatedly in parallel - and grow over time to offer more sophisticated invariant checking (a long running test that verifies that all pods can reach all targets). We should add the test harness as an e2e suite and add a release periodic and PR job for openshift/sdn and ovn-kube that allows us to trigger the test. We will have follow up work that ensures the tests are more effective, and probably want to look at process changes that ensure when we hit flakes that weren't caught by the stress test we add a new case. Things we can do later: 1. run an invariant checker (as a test, or part of the monitor, or as a disruptive test) that verifies the environment is currently working (exec into a pod and make sure it can curl masters over network, known working pods over service, directly reach pods over pod network, or access the host they are running on via host network) 2. add new types of e2e tests that try disruptive style things (delete the ovn-kube pod on a node and verify no failures are detected) 3. better instrument upgrades to verify that no connections are dropped, like we do for service load balancers 4. ... ?
For right now getting an openshift-sdn and ovn-kube stress test release job is the primary goal, then we will assess further.
hi, Mike there is new added stress test tools, I'm not sure if you notice this. assign this bug to you. thanks.
Verified with openshift-tests from registry.svc.ci.openshift.org/ocp/4.4-art-latest-2020-01-15-102334:tests. Running openshift/network/stress runs the network tests 15 times. I saw 45 failures (15 x 3 failing tests) which I will file bugs for separately.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581