Bug 1473370 - ResourceQuota controller observed making excessive LIST calls at scale
ResourceQuota controller observed making excessive LIST calls at scale
Status: VERIFIED
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity medium
: ---
: 3.7.0
Assigned To: Derek Carr
DeShuai Ma
:
Depends On: 1473042
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-20 11:59 EDT by Eric Paris
Modified: 2017-09-28 02:36 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: too many LIST calls were made by resource quota controller Consequence: this can impact performance of clusters at largest scales. Fix: reduced LIST calls made by resource quota controller by using shared informer caches. Result: LIST operations made to the master were reduced and information was pulled from a shared cache in the controller.
Story Points: ---
Clone Of: 1473042
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 2 DeShuai Ma 2017-09-28 02:36:11 EDT
Do some regression on openshift v3.7.0-0.127.0, quota works well, verify the bug

[root@ip-172-18-9-60 ~]# oc create quota quota --hard=services=10,secrets=10
resourcequota "quota" created
[root@ip-172-18-9-60 ~]# oc get quota
NAME      AGE
quota     8s
[root@ip-172-18-9-60 ~]# oc describe quota
Name:		quota
Namespace:	dma
Resource	Used	Hard
--------	----	----
secrets		9	10
services	0	10
[root@ip-172-18-9-60 ~]# oc get secrets
NAME                       TYPE                                  DATA      AGE
builder-dockercfg-rqbbz    kubernetes.io/dockercfg               1         40s
builder-token-4f1w2        kubernetes.io/service-account-token   4         40s
builder-token-vtnk7        kubernetes.io/service-account-token   4         40s
default-dockercfg-8kmtq    kubernetes.io/dockercfg               1         40s
default-token-02cf2        kubernetes.io/service-account-token   4         40s
default-token-kftvq        kubernetes.io/service-account-token   4         40s
deployer-dockercfg-p008f   kubernetes.io/dockercfg               1         40s
deployer-token-g3jqd       kubernetes.io/service-account-token   4         40s
deployer-token-lggtl       kubernetes.io/service-account-token   4         40s
[root@ip-172-18-9-60 ~]# oc get svc | wc -l
No resources found.
0
[root@ip-172-18-9-60 ~]# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/services/ExternalSvc.yaml
service "my-svc" created
[root@ip-172-18-9-60 ~]# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/secrets/secret.yaml
secret "test-secret" created
[root@ip-172-18-9-60 ~]# oc get secrets | grep -v NAME | wc -l
10
[root@ip-172-18-9-60 ~]# oc get svc |grep -v NAME | wc -l
1
[root@ip-172-18-9-60 ~]# oc describe quota
Name:		quota
Namespace:	dma
Resource	Used	Hard
--------	----	----
secrets		10	10
services	1	10
[root@ip-172-18-9-60 ~]# oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/secrets/secret-datastring-image.json
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/secrets/secret-datastring-image.json": secrets "secret-datastring-image" is forbidden: exceeded quota: quota, requested: secrets=1, used: secrets=10, limited: secrets=10

Note You need to log in before you can comment on or make changes to this bug.