Bug 2035302

Summary:	Concurrent bulk requests make neutron-server use 100% CPU on all the workers
Product:	Red Hat OpenStack	Reporter:	Slawek Kaplonski <skaplons>
Component:	openstack-neutron	Assignee:	Slawek Kaplonski <skaplons>
Status:	CLOSED ERRATA	QA Contact:	Itzik Brown <itbrown>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	ccamposr, chrisw, ekuris, itbrown, mdemaced, mdulko, rcernin, scohen
Target Milestone:	z9	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-15.2.1-1.20220120083333.40d217c.el8ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:	2024690	Environment:
Last Closed:	2022-12-07 20:28:59 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2024690
Bug Blocks:

Comment 10 Itzik Brown 2022-11-10 08:52:57 UTC

Verified with:
OSP RHOS-16.1-RHEL-8-20221108.n.1
OCP 4.11.0-0.nightly-2022-11-08-222031

Created 10 namespaces with 10 pods in each using a deployment.
Neutron was accessible and no 504 errors from Neutron were found in the Kuryr controller log

Comment 11 Itzik Brown 2022-11-10 15:39:16 UTC

So when running cloud density with WORKLOAD=cluster-density; export JOB_ITERATIONS=40
I saw errors in the logs(More than 1500 each):
2022-11-10 12:38:16.986 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://10.46.44.140:13696/v2.0/ports, 504 Gateway Time-out: The server didn't respond in time.
and
2022-11-10 12:35:10.973 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://10.46.44.140:13696/v2.0/trunks/12c3c4b5-3433-475d-bd38-6f95ada9f281/add_subports, 504 Gateway Time-out: The server didn't respond in time.
But Neutron seems to be responsive

Comment 18 errata-xmlrpc 2022-12-07 20:28:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack Platform 16.1.9 (openstack-neutron) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8870