Bug 2024690 - Concurrent bulk requests make neutron-server use 100% CPU on all the workers
Summary: Concurrent bulk requests make neutron-server use 100% CPU on all the workers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Slawek Kaplonski
QA Contact: Candido Campos
URL:
Whiteboard:
Depends On:
Blocks: 2035302
TreeView+ depends on / blocked
 
Reported: 2021-11-18 17:00 UTC by Michał Dulko
Modified: 2024-12-20 21:37 UTC (History)
5 users (show)

Fixed In Version: openstack-neutron-15.3.5-2.20220113150030.94e4cbb.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2035302 (view as bug list)
Environment:
Last Closed: 2022-03-23 22:12:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1954763 0 None None None 2021-12-14 11:39:23 UTC
Red Hat Issue Tracker OSP-10861 0 None None None 2021-11-18 17:01:48 UTC
Red Hat Product Errata RHBA-2022:1001 0 None None None 2022-03-23 22:12:49 UTC

Description Michał Dulko 2021-11-18 17:00:03 UTC 
Description of problem:
We've discovered this while trying to work out scale tests with Kuryr. When creating more pods in OpenShift Kuryr will shoot a bunch of bulk port create requests. We've see issues with attempts to create ~300 ports in bulks of 10 in 10 subnets.

Version-Release number of selected component (if applicable):
16.2_20210811.1, standalone (I know, it's not supported, but I still think some issue is there)

How reproducible:
Pretty much always.

Steps to Reproduce:
1. Install OpenShift with Kuryr on top of OpenStack.
2. Create 10 namespaces with 10 pods in each.
3. Observe neutron-server behavior, start seeing 504 responses to port POST requests sent to Neutron by kuryr-controller.

I'm trying to find a smaller reproducer.

Actual results:
Neutron stops to be responsive.

Expected results:
Neutron survive.

Additional info:
I'm happy to provide a cluster for debugging this.
Comment 16 errata-xmlrpc 2022-03-23 22:12:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1001
Note You need to log in before you can comment on or make changes to this bug.