Bug 1810501

Summary: Kuryr quota calculations are inefficient
Product: OpenShift Container Platform Reporter: Michał Dulko <mdulko>
Component: NetworkingAssignee: Michał Dulko <mdulko>
Networking sub component: kuryr QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: juriarte, ltomasbo
Version: 4.4   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1810591 (view as bug list) Environment:
Last Closed: 2020-08-04 18:03:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1819129    
Bug Blocks: 1810591    

Description Michał Dulko 2020-03-05 11:50:05 UTC 
Description of problem:
Currently when checking if there's still free quota for resources it's listing all those resources. This is highly inefficient, especially on bigger clouds. Kuryr should use quota_details Neutron exception instead to just query for the numbers maintained by Neutron itself.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Call Kuryr's readiness endpoint.
2. See that a call to list resources is made (note that it's cached for 120 seconds).

This isn't a great way to verify this, please contact me (mdulko) when verifying and I'll try to come up with a better solution.

Actual results:


Expected results:


Additional info:
Comment 3 Jon Uriarte 2020-04-03 07:58:52 UTC 
Verified in 4.5.0-0.nightly-2020-04-02-0043212 on top of OSP 16 RHOS_TRUNK-16.0-RHEL-8-20200324.n.0 compose.

Checked tenant quotas.

$ openstack quota list --network --detail --project shiftstack
+----------------------+--------+----------+-------+
| Resource             | In Use | Reserved | Limit |
+----------------------+--------+----------+-------+
| floating_ips         |      1 |        0 |    50 |
| networks             |     60 |        0 |   250 |
| ports                |    467 |        0 |  1500 |
| rbac_policies        |      0 |        0 |    10 |
| routers              |      1 |        0 |    10 |
| subnets              |     60 |        0 |   250 |
| subnet_pools         |      1 |        0 |    -1 |
| security_group_rules |    291 |        0 |  1000 |
| security_groups      |     71 |        0 |   250 |
+----------------------+--------+----------+-------+

When limiting the quota to the currently used quota there will be no more quota available
for resource creation. It will be detected by Kury-controller and set as Unhealthy due to
readiness probe failure.

ports quota:
-----------

(overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 467 shiftstack                                                                                                                                     
(overcloud) [stack@undercloud-0 ~]$ openstack quota list --network --detail --project shiftstack
+----------------------+--------+----------+-------+
| Resource             | In Use | Reserved | Limit |
+----------------------+--------+----------+-------+
| floating_ips         |      1 |        0 |    50 |
| networks             |     60 |        0 |   250 |
| ports                |    467 |        0 |   467 |
| rbac_policies        |      0 |        0 |    10 |
| routers              |      1 |        0 |    10 |
| subnets              |     60 |        0 |   250 |
| subnet_pools         |      1 |        0 |    -1 |
| security_group_rules |    291 |        0 |  1000 |
| security_groups      |     71 |        0 |   250 |
+----------------------+--------+----------+-------+


2020-04-03 07:50:11.717 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: ports
2020-04-03 07:50:11.719 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:50:11] "GET /ready HTTP/1.1" 500 -

Events:
  Type     Reason     Age                 From                            Message
  ----     ------     ----                ----                            -------
  Warning  Unhealthy  15s (x4 over 11h)  kubelet, ostest-xk585-master-0  Readiness probe failed: HTTP probe failed with statuscode: 500

(overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 1500 shiftstack                                                                                                                                    

2020-04-03 07:51:41.768 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-04-03 07:51:41.771 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:51:41] "GET /ready HTTP/1.1" 200 -


subnets quota:
-------------
$ openstack quota set --subnets 60 shiftstack

2020-04-03 07:53:11.626 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: subnets
2020-04-03 07:53:11.628 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:53:11] "GET /ready HTTP/1.1" 500 -


$ openstack quota set --subnets 250 shiftstack

2020-04-03 07:53:41.970 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-04-03 07:53:41.973 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:53:41] "GET /ready HTTP/1.1" 200 -


networks quota:
--------------
$ openstack quota set --networks 60 shiftstack

2020-04-03 07:55:11.733 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: networks
2020-04-03 07:55:11.737 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:55:11] "GET /ready HTTP/1.1" 500 -

$ openstack quota set --networks 250 shiftstack

2020-04-03 07:55:41.702 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-04-03 07:55:41.709 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:55:41] "GET /ready HTTP/1.1" 200 -


sec groups quota:
----------------
$ openstack quota set --secgroups 71 shiftstack

2020-04-03 07:56:11.984 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: security_groups
2020-04-03 07:56:11.986 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:56:11] "GET /ready HTTP/1.1" 500 -


$ openstack quota set --secgroups 250 shiftstack

2020-04-03 07:56:42.489 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:56:42] "GET /alive HTTP/1.1" 200 -
2020-04-03 07:56:42.774 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
Comment 5 errata-xmlrpc 2020-08-04 18:03:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5 image release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409