Bug 2088444

Summary: Under heavy load, kuryr create too many networks and/or subnets
Product: OpenShift Container Platform Reporter: rdobosz
Component: NetworkingAssignee: rdobosz
Networking sub component: kuryr QA Contact: Itzik Brown <itbrown>
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: medium CC: itbrown, mdemaced, mdulko, stephenfin
Version: 4.11Keywords: Triaged
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Neutron API instability and unreliability under load. Consequence: Kuryr can create duplicated networks and subnets for namespaces. Fix: Kuryr will periodically monitor Neutron for duplicate resources and attempt to delete them. Result: Duplicated resources get eventually deleted.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-29 14:26:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description rdobosz 2022-05-19 13:04:59 UTC 
Description of problem:

Kuryr sometimes creates Subnets leftovers, especially, when Neutron is under a heavy load. We have seen situations where a Namespace Network ends up with multiple Subnets.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
There should be no surplus accidentally created resources

Additional info:
Comment 3 Itzik Brown 2022-06-28 09:53:56 UTC 
Version
OCP 4.11.0-0.nightly-2022-06-25-132614
OSP RHOS-16.2-RHEL-8-20220610.n.1

1. Created 101 projects with a deployment of 5 replicas each

$ for i in $(seq 500 600); do oc new-project demo$i; oc create deployment  --image quay.io/kuryr/demo --replicas 5 demo; done

2. Checked  that there are not duplicated networks with the same name
openstack network list |grep demo | awk '{print $4}' |sed -e 's/.*\///' | sort | uniq -c  | awk '{print $1}' | grep -v 1

There were none

3. Checked that there are not duplicated subnets

openstack subnet list |grep demo | awk '{print $4}' |sed -e 's/.*\///' | sort | uniq -c  | awk '{print $1}' | grep -v 1
...
There are 

2 demo592
1 demo593
1 demo594
1 demo596
3 demo597
Comment 5 Maysa Macedo 2022-06-29 14:26:51 UTC 
Since this is a Neutron issue[1] and we already have a clean-up thread that removed leftover Subnets, I'm closing this bugzilla.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2083245