Bug 2035333

Summary:	Kuryr orphans ports on 504 errors from Neutron
Product:	OpenShift Container Platform	Reporter:	Michał Dulko <mdulko>
Component:	Networking	Assignee:	Maysa Macedo <mdemaced>
Networking sub component:	kuryr	QA Contact:	Itzik Brown <itbrown>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	itbrown, juriarte, mdemaced
Version:	4.10
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-12 04:40:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Michał Dulko 2021-12-23 16:10:01 UTC

Description of problem:
When we're creating many pods (~500 pods in 10 namespaces was a good benchmark numer) we're stressing Neutron and due to bug [1] the create ports requests will take enormous amount of time. If it goes over 2 minutes the HAProxy in front of Neutron API will time out and we'll get 504 error. This means that the ports are still being created on the Neutron side, but Kuryr will never get to tag them.

While ultimately a Neutron issue and a fix is proposed in [1], clients will most likely stick with problematic versions for a while, so we need a fix in Kuryr.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2024690

Version-Release number of selected component (if applicable): Current 16.1 and 16.2 are affected.


How reproducible: Most of the times.


Steps to Reproduce:
1. Modify https://github.com/thekubeworld/k8devel/blob/main/examples/pod/createNpods/parallel/createNpods.go to create just 50 pods per namespace.
2. Run it against a Kuryr OCP cluster.
3. Look in Kuryr logs for 504 responses from Neutron.

Actual results:
We get these 504 and a bunch of untagged ports (probably getting deleted when namespace gets deleted).

Expected results:
Kuryr limits Neutron requests and we don't get these.

Additional info:

Comment 5 Itzik Brown 2022-01-10 13:37:37 UTC

Ran https://github.com/thekubeworld/k8devel/blob/main/examples/pod/createNpods/parallel/createNpods.go (Adjusted to 5 namespaces) 4.10.0-0.nightly-2022-01-05-181126

Pods were stuck in ContainerCreating and the Kuryr controller was restarted several times without getting to a ready state.

Comment 7 Maysa Macedo 2022-01-19 14:12:15 UTC

I'm returning this bug back to ON_QA given that I tried multiple times to reproduce the issue mentioned both in fully virtual OSP env and in a hybrid env.
The only 504 traceback seen in the logs were related to the add_subports requests, which is not as problematic as the create_port requests pointed in this bz as the Neutron Ports are deleted when this request fails.

Comment 8 Itzik Brown 2022-01-19 14:15:50 UTC

It works with 4.10.0-0.nightly-2022-01-17-223655

Comment 10 ShiftStack Bugwatcher 2022-03-05 07:07:28 UTC

Removing the Triaged keyword because:
* the QE automation assessment (flag qe_test_coverage) is missing

Comment 12 errata-xmlrpc 2022-03-12 04:40:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056