Bug 1631182 - kuryr-controller keep restarting after the networks of k8s tenant exceeds quota
Summary: kuryr-controller keep restarting after the networks of k8s tenant exceeds quota
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-kuryr-kubernetes
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 14.0 (Rocky)
Assignee: Maysa Macedo
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-20 07:38 UTC by Itzik Brown
Modified: 2019-04-30 17:47 UTC (History)
8 users (show)

Fixed In Version: openstack-kuryr-kubernetes-0.6.2-0.20190305141049.a019712.el8ost
Doc Type: Bug Fix
Doc Text:
Previously, when the number of networks in the Kubernetes project exceeded the quota, the kuryr-controller pod would restart indefinitely due to having been marked as unhealthy. Now, a new readiness check validates the tenant's quota against the available Neutron resources. If the quota is reached, the controller pod is marked as 'Not Ready' and an action is required from the tenant side to increase the quota value or delete resources.
Clone Of:
Environment:
Last Closed: 2019-04-30 17:47:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 618655 0 None master: MERGED kuryr-kubernetes: Add quota readiness check to controller (If4d42f866d2d64cae63736f4c206bedca039258b) 2019-04-16 10:17:58 UTC
OpenStack gerrit 620883 0 None master: MERGED kuryr-kubernetes: Fix quota readiness check (I0a8a7665b01ca6f66c13aa71df0e74adca811d9b) 2019-04-16 10:17:54 UTC
Red Hat Product Errata RHBA-2019:0944 0 None None None 2019-04-30 17:47:49 UTC

Description Itzik Brown 2018-09-20 07:38:56 UTC
Description of problem:
When the number of networks in the k8s project exceeds the quota the kuryr-controller pod restarts indefinitely.

A pod can't be created in a namespace but a pod in the default namespace can.

When networks are deleted so the number of networks is again within the quota limits the kuryr-controller pod stops restarting.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.As described in the description
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Antoni Segura Puimedon 2018-09-20 09:02:13 UTC
What would be your expected behavior?

Comment 2 Michał Dulko 2018-09-26 09:56:54 UTC
IMO the correct behavior should be as follows:

1. Liveness probe shouldn't be failing. Service is up, though unhealthy.
2. Readiness probe should check the quotas and signal if we have 0 space left with any resource we need.

My only concern is that it seems to be impossible to give any message through the healthcheck mechanism, which is suboptimal.

Comment 3 Itzik Brown 2018-10-22 11:08:35 UTC
I think that Michal's comment should satisfy the requirement.

Comment 21 errata-xmlrpc 2019-04-30 17:47:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0944


Note You need to log in before you can comment on or make changes to this bug.