Description of problem: When we're creating many pods (~500 pods in 10 namespaces was a good benchmark numer) we're stressing Neutron and due to bug [1] the create ports requests will take enormous amount of time. If it goes over 2 minutes the HAProxy in front of Neutron API will time out and we'll get 504 error. This means that the ports are still being created on the Neutron side, but Kuryr will never get to tag them. While ultimately a Neutron issue and a fix is proposed in [1], clients will most likely stick with problematic versions for a while, so we need a fix in Kuryr. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024690 Version-Release number of selected component (if applicable): Current 16.1 and 16.2 are affected. How reproducible: Most of the times. Steps to Reproduce: 1. Modify https://github.com/thekubeworld/k8devel/blob/main/examples/pod/createNpods/parallel/createNpods.go to create just 50 pods per namespace. 2. Run it against a Kuryr OCP cluster. 3. Look in Kuryr logs for 504 responses from Neutron. Actual results: We get these 504 and a bunch of untagged ports (probably getting deleted when namespace gets deleted). Expected results: Kuryr limits Neutron requests and we don't get these. Additional info:
Ran https://github.com/thekubeworld/k8devel/blob/main/examples/pod/createNpods/parallel/createNpods.go (Adjusted to 5 namespaces) 4.10.0-0.nightly-2022-01-05-181126 Pods were stuck in ContainerCreating and the Kuryr controller was restarted several times without getting to a ready state.
I'm returning this bug back to ON_QA given that I tried multiple times to reproduce the issue mentioned both in fully virtual OSP env and in a hybrid env. The only 504 traceback seen in the logs were related to the add_subports requests, which is not as problematic as the create_port requests pointed in this bz as the Neutron Ports are deleted when this request fails.
It works with 4.10.0-0.nightly-2022-01-17-223655
Removing the Triaged keyword because: * the QE automation assessment (flag qe_test_coverage) is missing
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056