Bug 2088444 - Under heavy load, kuryr create too many networks and/or subnets
Summary: Under heavy load, kuryr create too many networks and/or subnets
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.11
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.11.0
Assignee: rdobosz
QA Contact: Itzik Brown
Depends On:
TreeView+ depends on / blocked
Reported: 2022-05-19 13:04 UTC by rdobosz
Modified: 2022-07-04 07:19 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Neutron API instability and unreliability under load. Consequence: Kuryr can create duplicated networks and subnets for namespaces. Fix: Kuryr will periodically monitor Neutron for duplicate resources and attempt to delete them. Result: Duplicated resources get eventually deleted.
Clone Of:
Last Closed: 2022-06-29 14:26:51 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 674 0 None open Bug 2088444: Add leftover resources protection 2022-06-21 11:21:04 UTC

Description rdobosz 2022-05-19 13:04:59 UTC
Description of problem:

Kuryr sometimes creates Subnets leftovers, especially, when Neutron is under a heavy load. We have seen situations where a Namespace Network ends up with multiple Subnets.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:
There should be no surplus accidentally created resources

Additional info:

Comment 3 Itzik Brown 2022-06-28 09:53:56 UTC
OCP 4.11.0-0.nightly-2022-06-25-132614
OSP RHOS-16.2-RHEL-8-20220610.n.1

1. Created 101 projects with a deployment of 5 replicas each

$ for i in $(seq 500 600); do oc new-project demo$i; oc create deployment  --image quay.io/kuryr/demo --replicas 5 demo; done

2. Checked  that there are not duplicated networks with the same name
openstack network list |grep demo | awk '{print $4}' |sed -e 's/.*\///' | sort | uniq -c  | awk '{print $1}' | grep -v 1

There were none

3. Checked that there are not duplicated subnets

openstack subnet list |grep demo | awk '{print $4}' |sed -e 's/.*\///' | sort | uniq -c  | awk '{print $1}' | grep -v 1
There are 

2 demo592
1 demo593
1 demo594
1 demo596
3 demo597

Comment 5 Maysa Macedo 2022-06-29 14:26:51 UTC
Since this is a Neutron issue[1] and we already have a clean-up thread that removed leftover Subnets, I'm closing this bugzilla.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2083245

Note You need to log in before you can comment on or make changes to this bug.