Red Hat Bugzilla – Bug 1400200
ClusterResourceQuota status total doesn't match sum of namespaces
Last modified: 2016-12-07 15:59:52 EST
Description of problem: (spawned from https://github.com/openshift/origin/issues/11560) When rapidly updating multiple namespaces controlled by a single clusterresourcequota, the status.total.used can get out of sync with the sum of the status.namespaces[*].used Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: oc delete project foo bar oc delete clusterresourcequota --all oc create clusterquota for-user-by-annotation --project-annotation-selector=openshift.io/requester=user --hard=secrets=50 oc new-project foo --as=user oc new-project bar --as=user oc get clusterresourcequota/for-user-by-annotation -o yaml Run that repeatedly until the status.total.used count is observed to not match sum(status.namespaces[*].used) Actual results: Expected results: Additional info:
fixed in origin 1.4.0 by https://github.com/openshift/origin/pull/11595/ fixed in ose >= 3.4.0.17 by https://github.com/openshift/ose/commit/c37bec075312db6c1399fbaa7a975787922f5dbe will be fixed in origin 1.3.x by https://github.com/openshift/origin/pull/12067/ will be fixed in ose 3.3.x by https://github.com/openshift/ose/pull/484
Tested on OCP 3.3.1.7(openshift v3.3.1.7, kubernetes v1.3.0+52492b4, etcd 2.3.0+git), delete a namespace which can be counted by clusterresourcequota, then get the clusterresourcequota info, resources under that namespace release(here status.total.used.secrets decrease by 9) and can be used again, but a little problem is that namespace info still remains. Here are steps 1. Create a clusterresourcequota resource with project-annotation-selector. # oc create clusterresourcequota crq-1 --project-annotation-selector=openshift.io/requester=user --hard=secrets=50,pods=20 clusterresourcequota "crq-1" created 2. Create namespaces with annotation matched. # oc new-project p-1 --as=user # oc new-project p-2 --as=user # oc get project | grep p- p-1 Active p-2 Active # oc get clusterresourcequota crq-1 -o json { "kind": "ClusterResourceQuota", "apiVersion": "v1", "metadata": { "name": "crq-1", "selfLink": "/oapi/v1/clusterresourcequotas/crq-1", "uid": "3a9505a7-b85f-11e6-b258-0ac1389dbd42", "resourceVersion": "67556", "creationTimestamp": "2016-12-02T07:16:22Z" }, "spec": { "selector": { "labels": null, "annotations": { "openshift.io/requester": "user" } }, "quota": { "hard": { "pods": "20", "secrets": "50" } } }, "status": { "total": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "18" } }, "namespaces": [ { "namespace": "p-1", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "9" } } }, { "namespace": "p-2", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "9" } } } ] } } 3. Delete a namespace. # oc delete project p-1 # oc get project | grep p- p-2 Active 4. Check clusterresourcequota info, p-1 namespace is there. # oc get clusterresourcequota crq-1 -o json { "kind": "ClusterResourceQuota", "apiVersion": "v1", "metadata": { "name": "crq-1", "selfLink": "/oapi/v1/clusterresourcequotas/crq-1", "uid": "3a9505a7-b85f-11e6-b258-0ac1389dbd42", "resourceVersion": "68388", "creationTimestamp": "2016-12-02T07:16:22Z" }, "spec": { "selector": { "labels": null, "annotations": { "openshift.io/requester": "user" } }, "quota": { "hard": { "pods": "20", "secrets": "50" } } }, "status": { "total": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "9" } }, "namespaces": [ { "namespace": "p-1", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "0" } } }, { "namespace": "p-2", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "9" } } } ] } } 5. Create a namespace with annotation matched again. # oc new-project p-4 --as=user # oc get project | grep p- p-2 Active p-4 Active # oc get clusterresourcequota crq-1 -o json { "kind": "ClusterResourceQuota", "apiVersion": "v1", "metadata": { "name": "crq-1", "selfLink": "/oapi/v1/clusterresourcequotas/crq-1", "uid": "3a9505a7-b85f-11e6-b258-0ac1389dbd42", "resourceVersion": "75173", "creationTimestamp": "2016-12-02T07:16:22Z" }, "spec": { "selector": { "labels": null, "annotations": { "openshift.io/requester": "user" } }, "quota": { "hard": { "pods": "20", "secrets": "50" } } }, "status": { "total": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "18" } }, "namespaces": [ { "namespace": "p-1", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "0" } } }, { "namespace": "p-2", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "9" } } }, { "namespace": "p-4", "status": { "hard": { "pods": "20", "secrets": "50" }, "used": { "pods": "0", "secrets": "9" } } } ] } }
The deleted namespace not being removed is a separate issue, and will require a separate fix (in 3.3.x and 3.4.x). Opened https://bugzilla.redhat.com/show_bug.cgi?id=1401073 to track that This bug is a fix for the sum(status.namespaces.used) != total.used
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:2915