+++ This bug was initially created as a clone of Bug #1810501 +++ Description of problem: Currently when checking if there's still free quota for resources it's listing all those resources. This is highly inefficient, especially on bigger clouds. Kuryr should use quota_details Neutron exception instead to just query for the numbers maintained by Neutron itself. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Call Kuryr's readiness endpoint. 2. See that a call to list resources is made (note that it's cached for 120 seconds). This isn't a great way to verify this, please contact me (mdulko) when verifying and I'll try to come up with a better solution. Actual results: Expected results: Additional info:
Verified in 4.4.0-0.nightly-2020-03-16-124852 on top of OSP 13 13.0.11 2020-03-10.1 puddle. Checked tenant quotas. $ openstack quota list --network --detail --project shiftstack +----------------------+--------+----------+-------+ | Resource | In Use | Reserved | Limit | +----------------------+--------+----------+-------+ | subnets | 62 | 0 | 250 | | routers | 1 | 0 | 10 | | security_group_rules | 234 | 0 | 1000 | | subnet_pools | 1 | 0 | -1 | | security_groups | 61 | 0 | 250 | | rbac_policies | 0 | 0 | 10 | | floating_ips | 1 | 0 | 50 | | networks | 63 | 0 | 250 | | ports | 552 | 0 | 1500 | +----------------------+--------+----------+-------+ When limiting the quota to the currently used quota there will be no more quota available for resource creation. It will be detected by Kury-controller and set as Unhealthy due to readiness probe failure. ports quota: ----------- (overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 552 shiftstack (overcloud) [stack@undercloud-0 ~]$ openstack quota list --network --detail --project shiftstack +----------------------+--------+----------+-------+ | Resource | In Use | Reserved | Limit | +----------------------+--------+----------+-------+ | subnets | 62 | 0 | 250 | | routers | 1 | 0 | 10 | | security_group_rules | 234 | 0 | 1000 | | subnet_pools | 1 | 0 | -1 | | security_groups | 61 | 0 | 250 | | rbac_policies | 0 | 0 | 10 | | floating_ips | 1 | 0 | 50 | | networks | 63 | 0 | 250 | | ports | 552 | 0 | 552 | +----------------------+--------+----------+-------+ 2020-03-17 15:08:16.943 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: ports 2020-03-17 15:08:16.947 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:08:16] "GET /ready HTTP/1.1" 500 - Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 13s (x14 over 11h) kubelet, ostest-6wf97-master-1 Readiness probe failed: HTTP probe failed with statuscode: 500 (overcloud) [stack@undercloud-0 ~]$ openstack quota set --ports 1500 shiftstack 2020-03-17 15:13:16.923 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-03-17 15:13:16.924 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:13:16] "GET /ready HTTP/1.1" 200 - subnets quota: ------------- $ openstack quota set --subnets 62 shiftstack 2020-03-17 15:15:17.067 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: subnets 2020-03-17 15:15:17.069 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:15:17] "GET /ready HTTP/1.1" 500 - $ openstack quota set --subnets 250 shiftstack 2020-03-17 15:17:16.889 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-03-17 15:17:16.890 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:17:16] "GET /ready HTTP/1.1" 200 - networks quota: -------------- $ openstack quota set --networks 63 shiftstack 2020-03-17 15:20:46.931 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: networks 2020-03-17 15:20:46.932 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:20:46] "GET /ready HTTP/1.1" 500 - $ openstack quota set --networks 250 shiftstack 2020-03-17 15:21:46.903 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-03-17 15:21:46.910 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:21:46] "GET /ready HTTP/1.1" 200 - sec groups quota: ---------------- $ openstack quota set --secgroups 61 shiftstack 2020-03-17 15:23:46.957 1 ERROR kuryr_kubernetes.utils [-] Quota exceeded for resource: security_groups 2020-03-17 15:23:46.958 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:23:46] "GET /ready HTTP/1.1" 500 - $ openstack quota set --secgroups 250 shiftstack 2020-03-17 15:24:47.571 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2020-03-17 15:24:47.572 1 INFO werkzeug [-] 10.196.0.16 - - [17/Mar/2020 15:24:47] "GET /ready HTTP/1.1" 200 -
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581