Bug 2077167

Summary:	periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node-serial is permfailing - unidling tests
Product:	OpenShift Container Platform	Reporter:	Ben Parees <bparees>
Component:	Networking	Assignee:	jamo luhrsen <jluhrsen>
Networking sub component:	ovn-kubernetes	QA Contact:	Anurag saxena <anusaxen>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	medium
Priority:	low	CC:	cholman, ffernand, hongli, mmasters, rravaiol, sippy, wking
Version:	4.9
Target Milestone:	---
Target Release:	4.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	job=periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node-serial=all job=periodic-ci-openshift-release-master-nightly-4.9-e2e-azure-fips=all
Last Closed:	2022-05-27 16:51:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ben Parees 2022-04-20 19:44:48 UTC

job:
periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node-serial 

is permfailing frequently in CI, see testgrid results:
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node-serial

The consistent failure is in this test:


: [sig-network-edge][Feature:Idling] Unidling should handle many TCP connections by possibly dropping those over a certain bound [Serial] [Suite:openshift/conformance/serial] expand_less 	38s
{  fail [github.com/openshift/origin/test/extended/idling/idling.go:346]: Expected
    <int>: 0
to be >=
    <int>: 16}


So we are starting w/ network edge to investigate.  Note that this is the 3rd or 4th bug reported for this test being stuck in a permfailing situation, so we don't seem to be fixing it correctly, or making the fix stick.


sample job:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node-serial/1516527466993160192

Comment 1 Miciah Dashiel Butler Masters 2022-04-21 16:14:58 UTC

Setting low priority and blocker- as no severity was specified.

Comment 2 Ben Parees 2022-04-21 16:43:51 UTC

given that this is permfailing one of our CI jobs, i'd like to see the priority raised to at least medium

we see no failures for this in 4.8, and 4.10+, but it's 1.4% of all failures in 4.9, so something seems to be distinctly broken in 4.9 specifically.

https://search.ci.openshift.org/?search=Unidling+should+handle+many+TCP+connections+by+possibly+dropping+those+over+a+certain+bound&maxAge=48h&context=1&type=bug%2Bjunit&name=4.8&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

https://search.ci.openshift.org/?search=Unidling+should+handle+many+TCP+connections+by+possibly+dropping+those+over+a+certain+bound&maxAge=48h&context=1&type=bug%2Bjunit&name=4.9&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

https://search.ci.openshift.org/?search=Unidling+should+handle+many+TCP+connections+by+possibly+dropping+those+over+a+certain+bound&maxAge=48h&context=1&type=bug%2Bjunit&name=4.10&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

https://search.ci.openshift.org/?search=Unidling+should+handle+many+TCP+connections+by+possibly+dropping+those+over+a+certain+bound&maxAge=48h&context=1&type=bug%2Bjunit&name=4.11&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 4 Ben Parees 2022-04-25 14:14:51 UTC

also this is possibly a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1989169  ?  that bug seems to have gotten stuck after getting to POST

Comment 7 Ben Parees 2022-04-28 19:08:09 UTC

This is a pretty consistent failure in the 4.9 serial jobs, which is preventing people from backporting changes to the 4.9 release branches:

https://search.ci.openshift.org/?search=+Unidling+should+handle+many+TCP+connections+by+possibly+dropping+those&maxAge=168h&context=1&type=junit&name=4.9.*serial&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Can we please raise the priority on this?

Comment 10 Ben Parees 2022-05-19 13:39:08 UTC

Candace, it's consistently failing in this job:

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-nightly-4.9-e2e-aws-single-node-serial

https://search.ci.openshift.org/?search=Unidling+should+handle+many+TCP+connections+by+possibly+dropping+those+over+a+certain+bound&maxAge=48h&context=1&type=bug%2Bjunit&name=4.9-e2e-aws-single-node-serial&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

So it may be more specific to single node in particular, but it's the reason that job is never passing for us, and why it's important to be addressed (so that we can start getting signal from that job again)

Comment 19 jamo luhrsen 2022-05-27 16:51:18 UTC


*** This bug has been marked as a duplicate of bug 2003228 ***