Bug 1810591 - Kuryr quota calculations are inefficient
Summary: Kuryr quota calculations are inefficient
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.4.0
Assignee: Luis Tomas Bolivar
QA Contact: GenadiC
URL:
Whiteboard:
Depends On: 1810501
Blocks: 1815019
TreeView+ depends on / blocked
 
Reported: 2020-03-05 14:44 UTC by Michał Dulko
Modified: 2020-05-04 11:45 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1810501
: 1815019 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:44:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 180 0 None closed [release-4.4] Bug 1810591: Ensures accurate quota calculation during the readiness checks 2020-04-27 08:44:38 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:45:26 UTC

Description Michał Dulko 2020-03-05 14:44:30 UTC
+++ This bug was initially created as a clone of Bug #1810501 +++

Description of problem:
Currently when checking if there's still free quota for resources it's listing all those resources. This is highly inefficient, especially on bigger clouds. Kuryr should use quota_details Neutron exception instead to just query for the numbers maintained by Neutron itself.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Call Kuryr's readiness endpoint.
2. See that a call to list resources is made (note that it's cached for 120 seconds).

This isn't a great way to verify this, please contact me (mdulko) when verifying and I'll try to come up with a better solution.

Actual results:


Expected results:


Additional info:

Comment 3 Jon Uriarte 2020-03-17 15:28:38 UTC
Verified in 4.4.0-0.nightly-2020-03-16-124852 on top of OSP 13 13.0.11 2020-03-10.1 puddle.

Checked tenant quotas.

$ openstack quota list --network --detail --project shiftstack
+----------------------+--------+----------+-------+
| Resource             | In Use | Reserved | Limit |
+----------------------+--------+----------+-------+
| subnets              |     62 |        0 |   250 |
| routers              |      1 |        0 |    10 |
| security_group_rules |    234 |        0 |  1000 |
| subnet_pools         |      1 |        0 |    -1 |
| security_groups      |     61 |        0 |   250 |
| rbac_policies        |      0 |        0 |    10 |
| floating_ips         |      1 |        0 |    50 |
| networks             |     63 |        0 |   250 |
| ports                |    552 |        0 |  1500 |
+----------------------+--------+----------+-------+

When limiting the quota to the currently used quota there will be no more quota available
for resource creation. It will be detected by Kury-controller and set as Unhealthy due to
readiness probe failure.

ports quota:
-----------

(overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 552 shiftstack                                                                                                                                     
(overcloud) [stack@undercloud-0 ~]$ openstack quota list --network --detail --project shiftstack
+----------------------+--------+----------+-------+
| Resource             | In Use | Reserved | Limit |
+----------------------+--------+----------+-------+
| subnets              |     62 |        0 |   250 |
| routers              |      1 |        0 |    10 |
| security_group_rules |    234 |        0 |  1000 |
| subnet_pools         |      1 |        0 |    -1 |
| security_groups      |     61 |        0 |   250 |
| rbac_policies        |      0 |        0 |    10 |
| floating_ips         |      1 |        0 |    50 |
| networks             |     63 |        0 |   250 |
| ports                |    552 |        0 |   552 |
+----------------------+--------+----------+-------+


2020-03-17 15:08:16.943 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: ports
2020-03-17 15:08:16.947 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:08:16] "GET /ready HTTP/1.1" 500 -

Events:
  Type     Reason     Age                 From                            Message
  ----     ------     ----                ----                            -------
  Warning  Unhealthy  13s (x14 over 11h)  kubelet, ostest-6wf97-master-1  Readiness probe failed: HTTP probe failed with statuscode: 500

(overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 1500 shiftstack                                                                                                                                    

2020-03-17 15:13:16.923 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:13:16.924 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:13:16] "GET /ready HTTP/1.1" 200 -


subnets quota:
-------------
$ openstack quota set --subnets 62 shiftstack

2020-03-17 15:15:17.067 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: subnets
2020-03-17 15:15:17.069 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:15:17] "GET /ready HTTP/1.1" 500 -


$ openstack quota set --subnets 250 shiftstack

2020-03-17 15:17:16.889 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:17:16.890 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:17:16] "GET /ready HTTP/1.1" 200 -

networks quota:
--------------
$ openstack quota set --networks 63 shiftstack

2020-03-17 15:20:46.931 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: networks
2020-03-17 15:20:46.932 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:20:46] "GET /ready HTTP/1.1" 500 -

$ openstack quota set --networks 250 shiftstack

2020-03-17 15:21:46.903 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:21:46.910 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:21:46] "GET /ready HTTP/1.1" 200 -

sec groups quota:
----------------
$ openstack quota set --secgroups 61 shiftstack

2020-03-17 15:23:46.957 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: security_groups
2020-03-17 15:23:46.958 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:23:46] "GET /ready HTTP/1.1" 500 -


$ openstack quota set --secgroups 250 shiftstack

2020-03-17 15:24:47.571 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
2020-03-17 15:24:47.572 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:24:47] "GET /ready HTTP/1.1" 200 -

Comment 8 errata-xmlrpc 2020-05-04 11:44:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.