Bug 1913751 - add third-party network plugin test suite to openshift-tests
Summary: add third-party network plugin test suite to openshift-tests
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Dan Winship
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks: 1948233
TreeView+ depends on / blocked
 
Reported: 2021-01-07 14:06 UTC by Dan Winship
Modified: 2021-04-10 23:14 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1948233 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:50:58 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 25785 0 None open Bug 1913751: Add openshift/network/third-party suite, for CNI plugin conformance 2021-01-25 20:21:27 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:51:13 UTC

Description Dan Winship 2021-01-07 14:06:47 UTC
We want to change the third-party network plugin certification instructions to use openshift-tests rather than the upstream e2e.test.

To verify:
build origin from source or pull the "openshift-tests" image and run

  openshift-tests run openshift/network/third-party

and it should run about 60 tests, some of which are expected to fail under openshift-sdn. (I think if you ran them under ovn-kubernetes they'd all pass though... But anyway, it doesn't matter whether the tests pass or not, just that (a) it runs tests rather than saying "no such test suite", and (b) the number of passed+failed tests selected by this suite is somewhere around 60 right now.)

Comment 2 zhaozhanqi 2021-01-26 10:10:37 UTC
Run test on 4.7.0-0.nightly-2021-01-26-044139 with build origin from source on OVN cluster.

test result is about 53 cases, and 4 cases failed. 

[sig-arch] Monitor cluster while tests execute

Failing tests:

[sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to create a functioning NodePort service [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]

error: 5 fail, 48 pass, 0 skip (8m42s)

Comment 3 Dan Winship 2021-01-26 16:18:04 UTC
Weird... I just ran on AWS with ovn-kubernetes, 4.7.0-0.nightly-2021-01-26-082025, and got

  53 pass, 0 skip (8m7s)

So anyway, not sure why you got 5 failures (something is flaking maybe?) but it looks like the test suite is correct

Comment 4 zhaozhanqi 2021-01-27 06:39:08 UTC
Aha, I ran again on one US client, 53 pass. but 5 failed in Beijing client. So this should be Beijing network issue 

Flaky invariants:

[sig-arch] Monitor cluster while tests execute

53 pass, 0 skip (7m26s)


Move this bug to verified.

Comment 7 errata-xmlrpc 2021-02-24 15:50:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.