Bug 1913751

Summary:	add third-party network plugin test suite to openshift-tests
Product:	OpenShift Container Platform	Reporter:	Dan Winship <danw>
Component:	Networking	Assignee:	Dan Winship <danw>
Networking sub component:	openshift-sdn	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	aconstan, anbhat
Version:	4.7
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	1948233 (view as bug list)		Environment:
Last Closed:	2021-02-24 15:50:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1948233

Description Dan Winship 2021-01-07 14:06:47 UTC

We want to change the third-party network plugin certification instructions to use openshift-tests rather than the upstream e2e.test.

To verify:
build origin from source or pull the "openshift-tests" image and run

  openshift-tests run openshift/network/third-party

and it should run about 60 tests, some of which are expected to fail under openshift-sdn. (I think if you ran them under ovn-kubernetes they'd all pass though... But anyway, it doesn't matter whether the tests pass or not, just that (a) it runs tests rather than saying "no such test suite", and (b) the number of passed+failed tests selected by this suite is somewhere around 60 right now.)

Comment 2 zhaozhanqi 2021-01-26 10:10:37 UTC

Run test on 4.7.0-0.nightly-2021-01-26-044139 with build origin from source on OVN cluster.

test result is about 53 cases, and 4 cases failed. 

[sig-arch] Monitor cluster while tests execute

Failing tests:

[sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]
[sig-network] Services should be able to create a functioning NodePort service [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]

error: 5 fail, 48 pass, 0 skip (8m42s)

Comment 3 Dan Winship 2021-01-26 16:18:04 UTC

Weird... I just ran on AWS with ovn-kubernetes, 4.7.0-0.nightly-2021-01-26-082025, and got

  53 pass, 0 skip (8m7s)

So anyway, not sure why you got 5 failures (something is flaking maybe?) but it looks like the test suite is correct

Comment 4 zhaozhanqi 2021-01-27 06:39:08 UTC

Aha, I ran again on one US client, 53 pass. but 5 failed in Beijing client. So this should be Beijing network issue 

Flaky invariants:

[sig-arch] Monitor cluster while tests execute

53 pass, 0 skip (7m26s)


Move this bug to verified.

Comment 7 errata-xmlrpc 2021-02-24 15:50:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633