Bug 2035333 - Kuryr orphans ports on 504 errors from Neutron
Summary: Kuryr orphans ports on 504 errors from Neutron
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.10.0
Assignee: Maysa Macedo
QA Contact: Itzik Brown
Depends On:
TreeView+ depends on / blocked
Reported: 2021-12-23 16:10 UTC by Michał Dulko
Modified: 2022-03-12 04:40 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2022-03-12 04:40:05 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 619 0 None open Bug 2035333: Limit the number of concurrent create_ports requests 2021-12-23 16:14:25 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:40:21 UTC

Description Michał Dulko 2021-12-23 16:10:01 UTC
Description of problem:
When we're creating many pods (~500 pods in 10 namespaces was a good benchmark numer) we're stressing Neutron and due to bug [1] the create ports requests will take enormous amount of time. If it goes over 2 minutes the HAProxy in front of Neutron API will time out and we'll get 504 error. This means that the ports are still being created on the Neutron side, but Kuryr will never get to tag them.

While ultimately a Neutron issue and a fix is proposed in [1], clients will most likely stick with problematic versions for a while, so we need a fix in Kuryr.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2024690

Version-Release number of selected component (if applicable): Current 16.1 and 16.2 are affected.

How reproducible: Most of the times.

Steps to Reproduce:
1. Modify https://github.com/thekubeworld/k8devel/blob/main/examples/pod/createNpods/parallel/createNpods.go to create just 50 pods per namespace.
2. Run it against a Kuryr OCP cluster.
3. Look in Kuryr logs for 504 responses from Neutron.

Actual results:
We get these 504 and a bunch of untagged ports (probably getting deleted when namespace gets deleted).

Expected results:
Kuryr limits Neutron requests and we don't get these.

Additional info:

Comment 5 Itzik Brown 2022-01-10 13:37:37 UTC
Ran https://github.com/thekubeworld/k8devel/blob/main/examples/pod/createNpods/parallel/createNpods.go (Adjusted to 5 namespaces) 4.10.0-0.nightly-2022-01-05-181126

Pods were stuck in ContainerCreating and the Kuryr controller was restarted several times without getting to a ready state.

Comment 7 Maysa Macedo 2022-01-19 14:12:15 UTC
I'm returning this bug back to ON_QA given that I tried multiple times to reproduce the issue mentioned both in fully virtual OSP env and in a hybrid env.
The only 504 traceback seen in the logs were related to the add_subports requests, which is not as problematic as the create_port requests pointed in this bz as the Neutron Ports are deleted when this request fails.

Comment 8 Itzik Brown 2022-01-19 14:15:50 UTC
It works with 4.10.0-0.nightly-2022-01-17-223655

Comment 10 ShiftStack Bugwatcher 2022-03-05 07:07:28 UTC
Removing the Triaged keyword because:
* the QE automation assessment (flag qe_test_coverage) is missing

Comment 12 errata-xmlrpc 2022-03-12 04:40:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.