Bug 1473370 - ResourceQuota controller observed making excessive LIST calls at scale
Summary: ResourceQuota controller observed making excessive LIST calls at scale
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.7.0
Assignee: Derek Carr
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On: 1473042
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-20 15:59 UTC by Eric Paris
Modified: 2017-11-28 22:04 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: too many LIST calls were made by resource quota controller Consequence: this can impact performance of clusters at largest scales. Fix: reduced LIST calls made by resource quota controller by using shared informer caches. Result: LIST operations made to the master were reduced and information was pulled from a shared cache in the controller.
Clone Of: 1473042
Environment:
Last Closed: 2017-11-28 22:04:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Comment 2 DeShuai Ma 2017-09-28 06:36:11 UTC
Do some regression on openshift v3.7.0-0.127.0, quota works well, verify the bug

[root@ip-172-18-9-60 ~]# oc create quota quota --hard=services=10,secrets=10
resourcequota "quota" created
[root@ip-172-18-9-60 ~]# oc get quota
NAME      AGE
quota     8s
[root@ip-172-18-9-60 ~]# oc describe quota
Name:		quota
Namespace:	dma
Resource	Used	Hard
--------	----	----
secrets		9	10
services	0	10
[root@ip-172-18-9-60 ~]# oc get secrets
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-rqbbz    kubernetes.io/dockercfg               1         40s
builder-token-4f1w2        kubernetes.io/service-account-token   4         40s
builder-token-vtnk7        kubernetes.io/service-account-token   4         40s
default-dockercfg-8kmtq    kubernetes.io/dockercfg               1         40s
default-token-02cf2        kubernetes.io/service-account-token   4         40s
default-token-kftvq        kubernetes.io/service-account-token   4         40s
deployer-dockercfg-p008f   kubernetes.io/dockercfg               1         40s
deployer-token-g3jqd       kubernetes.io/service-account-token   4         40s
deployer-token-lggtl       kubernetes.io/service-account-token   4         40s
[root@ip-172-18-9-60 ~]# oc get svc | wc -l
No resources found.
0
[root@ip-172-18-9-60 ~]# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/services/ExternalSvc.yaml
service "my-svc" created
[root@ip-172-18-9-60 ~]# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/secrets/secret.yaml
secret "test-secret" created
[root@ip-172-18-9-60 ~]# oc get secrets | grep -v NAME | wc -l
10
[root@ip-172-18-9-60 ~]# oc get svc |grep -v NAME | wc -l
1
[root@ip-172-18-9-60 ~]# oc describe quota
Name:		quota
Namespace:	dma
Resource	Used	Hard
--------	----	----
secrets		10	10
services	1	10
[root@ip-172-18-9-60 ~]# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/secrets/secret-datastring-image.json
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/secrets/secret-datastring-image.json": secrets "secret-datastring-image" is forbidden: exceeded quota: quota, requested: secrets=1, used: secrets=10, limited: secrets=10

Comment 5 errata-xmlrpc 2017-11-28 22:04:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.