Bug 1810591

Summary: Kuryr quota calculations are inefficient
Product: OpenShift Container Platform Reporter: Michał Dulko <mdulko>
Component: NetworkingAssignee: Luis Tomas Bolivar <ltomasbo>
Networking sub component: kuryr QA Contact: GenadiC <gcheresh>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: gcheresh, igor.tiunov, juriarte, mtaru
Version: 4.4   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1810501
: 1815019 (view as bug list) Environment:
Last Closed: 2020-05-04 11:44:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1810501    
Bug Blocks: 1815019    

Description Michał Dulko 2020-03-05 14:44:30 UTC 
+++ This bug was initially created as a clone of Bug #1810501 +++

Description of problem:
Currently when checking if there's still free quota for resources it's listing all those resources. This is highly inefficient, especially on bigger clouds. Kuryr should use quota_details Neutron exception instead to just query for the numbers maintained by Neutron itself.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Call Kuryr's readiness endpoint.
2. See that a call to list resources is made (note that it's cached for 120 seconds).

This isn't a great way to verify this, please contact me (mdulko) when verifying and I'll try to come up with a better solution.

Actual results:


Expected results:


Additional info:
Comment 3 Jon Uriarte 2020-03-17 15:28:38 UTC 
Verified in 4.4.0-0.nightly-2020-03-16-124852 on top of OSP 13 13.0.11 2020-03-10.1 puddle.

Checked tenant quotas.

$ openstack quota list --network --detail --project shiftstack
+----------------------+--------+----------+-------+
| Resource             | In Use | Reserved | Limit |
+----------------------+--------+----------+-------+
| subnets              |     62 |        0 |   250 |
| routers              |      1 |        0 |    10 |
| security_group_rules |    234 |        0 |  1000 |
| subnet_pools         |      1 |        0 |    -1 |
| security_groups      |     61 |        0 |   250 |
| rbac_policies        |      0 |        0 |    10 |
| floating_ips         |      1 |        0 |    50 |
| networks             |     63 |        0 |   250 |
| ports                |    552 |        0 |  1500 |
+----------------------+--------+----------+-------+

When limiting the quota to the currently used quota there will be no more quota available
for resource creation. It will be detected by Kury-controller and set as Unhealthy due to
readiness probe failure.

ports quota:
-----------

(overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 552 shiftstack                                                                                                                                     
(overcloud) [stack@undercloud-0 ~]$ openstack quota list --network --detail --project shiftstack
+----------------------+--------+----------+-------+
| Resource             | In Use | Reserved | Limit |
+----------------------+--------+----------+-------+
| subnets              |     62 |        0 |   250 |
| routers              |      1 |        0 |    10 |
| security_group_rules |    234 |        0 |  1000 |
| subnet_pools         |      1 |        0 |    -1 |
| security_groups      |     61 |        0 |   250 |
| rbac_policies        |      0 |        0 |    10 |
| floating_ips         |      1 |        0 |    50 |
| networks             |     63 |        0 |   250 |
| ports                |    552 |        0 |   552 |
+----------------------+--------+----------+-------+


2020-03-17 15:08:16.943 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: ports
2020-03-17 15:08:16.947 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:08:16] "GET /ready HTTP/1.1" 500 -

Events:
  Type     Reason     Age                 From                            Message
  ----     ------     ----                ----                            -------
  Warning  Unhealthy  13s (x14 over 11h)  kubelet, ostest-6wf97-master-1  Readiness probe failed: HTTP probe failed with statuscode: 500

(overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 1500 shiftstack                                                                                                                                    

2020-03-17 15:13:16.923 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:13:16.924 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:13:16] "GET /ready HTTP/1.1" 200 -


subnets quota:
-------------
$ openstack quota set --subnets 62 shiftstack

2020-03-17 15:15:17.067 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: subnets
2020-03-17 15:15:17.069 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:15:17] "GET /ready HTTP/1.1" 500 -


$ openstack quota set --subnets 250 shiftstack

2020-03-17 15:17:16.889 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:17:16.890 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:17:16] "GET /ready HTTP/1.1" 200 -

networks quota:
--------------
$ openstack quota set --networks 63 shiftstack

2020-03-17 15:20:46.931 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: networks
2020-03-17 15:20:46.932 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:20:46] "GET /ready HTTP/1.1" 500 -

$ openstack quota set --networks 250 shiftstack

2020-03-17 15:21:46.903 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:21:46.910 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:21:46] "GET /ready HTTP/1.1" 200 -

sec groups quota:
----------------
$ openstack quota set --secgroups 61 shiftstack

2020-03-17 15:23:46.957 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: security_groups
2020-03-17 15:23:46.958 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:23:46] "GET /ready HTTP/1.1" 500 -


$ openstack quota set --secgroups 250 shiftstack

2020-03-17 15:24:47.571 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:24:47.572 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:24:47] "GET /ready HTTP/1.1" 200 -
Comment 8 errata-xmlrpc 2020-05-04 11:44:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581