Description of problem: Currently when checking if there's still free quota for resources it's listing all those resources. This is highly inefficient, especially on bigger clouds. Kuryr should use quota_details Neutron exception instead to just query for the numbers maintained by Neutron itself. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Call Kuryr's readiness endpoint. 2. See that a call to list resources is made (note that it's cached for 120 seconds). This isn't a great way to verify this, please contact me (mdulko) when verifying and I'll try to come up with a better solution. Actual results: Expected results: Additional info:
Verified in 4.5.0-0.nightly-2020-04-02-0043212 on top of OSP 16 RHOS_TRUNK-16.0-RHEL-8-20200324.n.0 compose. Checked tenant quotas. $ openstack quota list --network --detail --project shiftstack +----------------------+--------+----------+-------+ | Resource | In Use | Reserved | Limit | +----------------------+--------+----------+-------+ | floating_ips | 1 | 0 | 50 | | networks | 60 | 0 | 250 | | ports | 467 | 0 | 1500 | | rbac_policies | 0 | 0 | 10 | | routers | 1 | 0 | 10 | | subnets | 60 | 0 | 250 | | subnet_pools | 1 | 0 | -1 | | security_group_rules | 291 | 0 | 1000 | | security_groups | 71 | 0 | 250 | +----------------------+--------+----------+-------+ When limiting the quota to the currently used quota there will be no more quota available for resource creation. It will be detected by Kury-controller and set as Unhealthy due to readiness probe failure. ports quota: ----------- (overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 467 shiftstack (overcloud) [stack@undercloud-0 ~]$ openstack quota list --network --detail --project shiftstack +----------------------+--------+----------+-------+ | Resource | In Use | Reserved | Limit | +----------------------+--------+----------+-------+ | floating_ips | 1 | 0 | 50 | | networks | 60 | 0 | 250 | | ports | 467 | 0 | 467 | | rbac_policies | 0 | 0 | 10 | | routers | 1 | 0 | 10 | | subnets | 60 | 0 | 250 | | subnet_pools | 1 | 0 | -1 | | security_group_rules | 291 | 0 | 1000 | | security_groups | 71 | 0 | 250 | +----------------------+--------+----------+-------+ 2020-04-03 07:50:11.717 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: ports 2020-04-03 07:50:11.719 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:50:11] "GET /ready HTTP/1.1" 500 - Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 15s (x4 over 11h) kubelet, ostest-xk585-master-0 Readiness probe failed: HTTP probe failed with statuscode: 500 (overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 1500 shiftstack 2020-04-03 07:51:41.768 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-04-03 07:51:41.771 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:51:41] "GET /ready HTTP/1.1" 200 - subnets quota: ------------- $ openstack quota set --subnets 60 shiftstack 2020-04-03 07:53:11.626 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: subnets 2020-04-03 07:53:11.628 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:53:11] "GET /ready HTTP/1.1" 500 - $ openstack quota set --subnets 250 shiftstack 2020-04-03 07:53:41.970 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-04-03 07:53:41.973 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:53:41] "GET /ready HTTP/1.1" 200 - networks quota: -------------- $ openstack quota set --networks 60 shiftstack 2020-04-03 07:55:11.733 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: networks 2020-04-03 07:55:11.737 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:55:11] "GET /ready HTTP/1.1" 500 - $ openstack quota set --networks 250 shiftstack 2020-04-03 07:55:41.702 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-04-03 07:55:41.709 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:55:41] "GET /ready HTTP/1.1" 200 - sec groups quota: ---------------- $ openstack quota set --secgroups 71 shiftstack 2020-04-03 07:56:11.984 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: security_groups 2020-04-03 07:56:11.986 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:56:11] "GET /ready HTTP/1.1" 500 - $ openstack quota set --secgroups 250 shiftstack 2020-04-03 07:56:42.489 1 INFO werkzeug [-] ::ffff:10.196.1.12 - - [03/Apr/2020 07:56:42] "GET /alive HTTP/1.1" 200 - 2020-04-03 07:56:42.774 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.5 image release advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409