Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1975476

Summary:	OVN jobs are flaking on [sig-network] Netpol [LinuxOnly] NetworkPolicy between server and client should deny egress from all pods in a namespace
Product:	OpenShift Container Platform	Reporter:	Stephen Benjamin <stbenjam>
Component:	Networking	Assignee:	Dan Winship <danw>
Networking sub component:	ovn-kubernetes	QA Contact:	Anurag saxena <anusaxen>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	high	CC:	anbhat, danw, sippy
Version:	4.9
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	[sig-network] Netpol [LinuxOnly] NetworkPolicy between server and client should deny egress from all pods in a namespace [Feature:NetworkPolicy] [Skipped:Network/OpenShiftSDN/Multitenant] [Skipped:Network/OpenShiftSDN] [Suite:openshift/conformance/parallel] [Suite:k8s]
Last Closed:	2021-08-03 13:59:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Stephen Benjamin 2021-06-23 18:24:44 UTC

test:
[sig-network] Netpol [LinuxOnly] NetworkPolicy between server and client should deny egress from all pods in a namespace [Feature:NetworkPolicy]  [Skipped:Network/OpenShiftSDN/Multitenant] [Skipped:Network/OpenShiftSDN] [Suite:openshift/conformance/parallel] [Suite:k8s] 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=4.9&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-network%5C%5D+Netpol+%5C%5BLinuxOnly%5C%5D+NetworkPolicy+between+server+and+client+should+stop+enforcing+policies+after+they+are+deleted+%5C%5BFeature%3ANetworkPolicy%5C%5D+%5C%5BSkipped%3ANetwork%2FOpenShiftSDN%2FMultitenant%5C%5D+%5C%5BSkipped%3ANetwork%2FOpenShiftSDN%5C%5D+%5C%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5C%5D+%5C%5BSuite%3Ak8s%5C%5D


This appears to be affecting many OVN jobs.Example jobs:

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-ovn-4.9/1407730987508436992

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/19614/rehearse-19614-pull-ci-openshift-router-release-4.9-e2e-metal-ipi-ovn-router/1407689225729478656

Comment 1 Dan Winship 2021-06-24 19:58:24 UTC

OK, this is timing again. The ValidateOrFail() function in test/e2e/network/netpol/test_helpers.go seems to be broken. It does a first try, and then, if it doesn't get the right result the first time, it retries the test. But it seems like it doesn't clean up correctly / merge the results correctly, so the retry has no effect; if it failed the first time (because the policies hadn't been programmed fully at that point) then the test will fail, even if it works the second time.

Comment 2 Dan Winship 2021-06-24 20:06:50 UTC

updated https://github.com/openshift/origin/pull/26266 (4.9) and https://github.com/openshift/origin/pull/26263 (4.8) to now skip the entire "Netpol" suite. We will still at least be running the "legacy" NetworkPolicy tests in 4.8.0 though.

Comment 3 Dan Winship 2021-06-25 14:09:30 UTC

(In reply to Dan Winship from comment #1)
> OK, this is timing again. The ValidateOrFail() function in
> test/e2e/network/netpol/test_helpers.go seems to be broken. It does a first
> try, and then, if it doesn't get the right result the first time, it retries
> the test. But it seems like it doesn't clean up correctly / merge the
> results correctly, so the retry has no effect; if it failed the first time
> (because the policies hadn't been programmed fully at that point) then the
> test will fail, even if it works the second time.

OK, that analysis seems to be wrong; ValidateOrFail does seem to correctly merge the results of the first and second tries when I test it. So I'm not sure exactly how it's failing in this case.

Comment 4 Gabe Montero 2021-07-19 17:21:14 UTC

In case it helps I believe I got an instance of this in https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26336/pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6/1417133688482697216

Comment 5 Dan Winship 2021-08-03 13:59:57 UTC


*** This bug has been marked as a duplicate of bug 1980141 ***