Bug 2035302 - Concurrent bulk requests make neutron-server use 100% CPU on all the workers
Summary: Concurrent bulk requests make neutron-server use 100% CPU on all the workers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Slawek Kaplonski
QA Contact: Itzik Brown
URL:
Whiteboard:
Depends On: 2024690
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-23 14:33 UTC by Slawek Kaplonski
Modified: 2022-12-07 20:29 UTC (History)
8 users (show)

Fixed In Version: openstack-neutron-15.2.1-1.20220120083333.40d217c.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 2024690
Environment:
Last Closed: 2022-12-07 20:28:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-11942 0 None None None 2021-12-23 14:42:08 UTC
Red Hat Product Errata RHSA-2022:8870 0 None None None 2022-12-07 20:29:44 UTC

Comment 10 Itzik Brown 2022-11-10 08:52:57 UTC 
Verified with:
OSP RHOS-16.1-RHEL-8-20221108.n.1
OCP 4.11.0-0.nightly-2022-11-08-222031

Created 10 namespaces with 10 pods in each using a deployment.
Neutron was accessible and no 504 errors from Neutron were found in the Kuryr controller log
Comment 11 Itzik Brown 2022-11-10 15:39:16 UTC 
So when running cloud density with WORKLOAD=cluster-density; export JOB_ITERATIONS=40
I saw errors in the logs(More than 1500 each):
2022-11-10 12:38:16.986 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://10.46.44.140:13696/v2.0/ports, 504 Gateway Time-out: The server didn't respond in time.
and
2022-11-10 12:35:10.973 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://10.46.44.140:13696/v2.0/trunks/12c3c4b5-3433-475d-bd38-6f95ada9f281/add_subports, 504 Gateway Time-out: The server didn't respond in time.
But Neutron seems to be responsive
Comment 18 errata-xmlrpc 2022-12-07 20:28:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack Platform 16.1.9 (openstack-neutron) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8870
Note You need to log in before you can comment on or make changes to this bug.