Description of problem: The resource quota statistics do not update dynamically even when the resource is deleted. Please see below tests: On OCP 4 cluster: ~~~~~~~~~~~~~~~~~~~~~~~~~ [quicklab@upi-0 ~]$ oc version Client Version: 4.4.18 Server Version: 4.4.18 Kubernetes Version: v1.17.1+20ba474 ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ [quicklab@upi-0 ~]$ oc new-app httpd-example warning: Cannot find git. Ensure that it is installed and in your path. Git is required to work with git repositories. --> Deploying template "openshift/httpd-example" to project test11 ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ [quicklab@upi-0 ~]$ oc describe quota openshift-object-counts Name: openshift-object-counts Namespace: test11 Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ [quicklab@upi-0 ~]$ oc delete all -l app=httpd-example replicationcontroller "httpd-example-1" deleted service "httpd-example" deleted deploymentconfig.apps.openshift.io "httpd-example" deleted ~~~~~~~~~~~~~~~~~~~~~~~~~~~ After an hour: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [quicklab@upi-0 ~]$ oc describe quota openshift-object-counts Name: openshift-object-counts Namespace: test11 Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 You have new mail in /var/spool/mail/quicklab ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ However, it's working in 3.11: 1. On OCP 3.X cluster ~~~~~~~~~~~~~~~~~~~~ [quicklab@master-0 ~]$ oc version Server https://openshift.internal.sthakare11.lab.pnq2.cee.redhat.com:443 openshift v3.11.272 kubernetes v1.11.0+d4cacc0 ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ [quicklab@master-0 ~]$ oc new-app httpd-example --> Deploying template "openshift/httpd-example" to project test12 ~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ [quicklab@master-0 ~]$ oc describe quota openshift-object-counts Name: openshift-object-counts Namespace: test12 Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~ [quicklab@master-0 ~]$ oc delete all -l app=httpd-example replicationcontroller "httpd-example-1" deleted service "httpd-example" deleted ~~~~~~~~~~~~~~~~ After some time: ~~~~~~~~~~~~~~~~ <pre>[quicklab@master-0 ~]$ oc describe quota openshift-object-counts Name: openshift-object-counts Namespace: test12 Resource Used Hard -------- ---- ---- openshift.io/imagestreams 0 10 </pre> ~~~~~~~~~~~~~~~~~ Version-Release number of selected component (if applicable): $ oc version Client Version: 4.4.19 Server Version: 4.4.19 Kubernetes Version: v1.17.1+6af3663 How reproducible: 1. Create a test project and application 2. Create a resource quota for the count statistics. doc[1] 3. Check the count statistics. 4. Delete the application and check the count statistics again Steps to Reproduce: 1. Create a test project and application 2. Create a resource quota for the count statistics. doc[1] 3. Check the count statistics. 4. Delete the application and check the count statistics again Actual results: Quota count statistics not updating after deleting the image stream. Expected results: Quota count statistics should reflect the changes dynamically. Additional info: [1] https://docs.openshift.com/container-platform/3.11/dev_guide/compute_resources.html#dev-viewing-quotas
Not sure if this is KCM or OCM, but sending to KCM because Quota. Could also be OCM if some sort of cleanup is failing to happen on imagestreams.
Seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=1874651 sending over to openshift-apiserver team
Maciej, i see that a customer is being effected as per the https://bugzilla.redhat.com/show_bug.cgi?id=1874651#c5, is it possible to have a fix for this in 4.6 ? Or based on your comment 2 should we close this bug as duplicate ?
(In reply to RamaKasturi from comment #3) > Maciej, i see that a customer is being effected as per the > https://bugzilla.redhat.com/show_bug.cgi?id=1874651#c5, is it possible to > have a fix for this in 4.6 ? Or based on your comment 2 should we close this > bug as duplicate ? I'm not 100% sure it's duplicate, although it seems similar I'll leave Lukasz that decision, before closing this as duplicate. Also with the current assignements, I doubt we'll be able to fix that right away. At most we might consider backporting the fix to 4.6.z, if needed.
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.
I was able to reproduce the issue today on a 4.7 cluster. I don't think it is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1874651. I'm trying to find the root cause of the issue.
The quota is not dynamically updated because the resource name is incorrect. It must be of the form of count/<resource>.<group> as described in https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota I tested it successfully with the following ResourceQuota: apiVersion: v1 kind: ResourceQuota metadata: name: openshift-object-counts spec: hard: count/imagestreams.image.openshift.io: "10" I did a few manual tests to see why it went up but not down once set to "openshift.io/imagestreams" It looks like the admission plugin in the openshift-apiserver had more info to properly resolve "openshift.io/imagestreams" so it went up during the object creation. I didn't go down because KCM wasn't unable to resolve "openshift.io/imagestreams". In general "openshift.io/imagestreams" is accepted because the validation only checks the name which is valid. I'm investigating this further to see what could be done to warn the end-user earlier.
Found it, the openshift-apiserver maintains an alias ("openshift.io/imagestreams") https://github.com/openshift/openshift-apiserver/blob/master/pkg/quota/image/registry.go#L16 That's why the quota was calculated on creation but wasn't' during removal. KCM doesn't know about this alias. I'm not sure what is our depreciation policy (I'm going to check that). I can imagine it can cause some issues during migration from 3.X to 4.X. In that case, we should update KCM so that it knows about the custom alias.
Verified bug with the payload below and i see that the fix works fine. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc version Client Version: 4.7.0-0.nightly-2020-11-10-023606 Server Version: 4.7.0-0.nightly-2020-11-10-023606 Kubernetes Version: v1.19.2+7e80e12 Below are the steps followed to verify the bug in 4.7: ======================================================= [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-project test Now using project "test" on server "https://api.knarra1110.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-app httpd-example --> Deploying template "openshift/httpd-example" to project test Apache HTTP Server --------- An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. The following service(s) have been created in your project: httpd-example. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. * With parameters: * Name=httpd-example * Namespace=openshift * Memory Limit=512Mi * Git Repository URL=https://github.com/sclorg/httpd-ex.git * Git Reference= * Context Directory= * Application Hostname= * GitHub Webhook Secret=0gil7sRyormCtAFXHeJSgYjbGWkBbPaeSQaQJVP3 # generated * Generic Webhook Secret=TGj6I2kRTEFwojU2M5eabqTPJeQNq2icpPTJW4vA # generated --> Creating resources ... service "httpd-example" created route.route.openshift.io "httpd-example" created imagestream.image.openshift.io "httpd-example" created buildconfig.build.openshift.io "httpd-example" created deploymentconfig.apps.openshift.io "httpd-example" created --> Success Access your application via route 'httpd-example-test.apps.knarra1110.qe.devcluster.openshift.com' Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress. Run 'oc status' to view your app. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 12s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 0/1 Completed 0 66s httpd-example-1-deploy 0/1 Completed 0 37s httpd-example-1-qctph 1/1 Running 0 35s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc create quota test1 --hard=openshift.io/imagestreams=10 resourcequota/test1 created [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: test Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc delete all -l app=httpd-example replicationcontroller "httpd-example-1" deleted service "httpd-example" deleted deploymentconfig.apps.openshift.io "httpd-example" deleted buildconfig.build.openshift.io "httpd-example" deleted imagestream.image.openshift.io "httpd-example" deleted route.route.openshift.io "httpd-example" deleted [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: test Resource Used Hard -------- ---- ---- openshift.io/imagestreams 0 10 Tried the same in 4.6 and i see issue is reproducible: ======================================================= [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-project knarra Now using project "knarra" on server "https://api.yinzhou-vs-6.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-app httpd-example --> Deploying template "openshift/httpd-example" to project knarra Apache HTTP Server --------- An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. The following service(s) have been created in your project: httpd-example. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. * With parameters: * Name=httpd-example * Namespace=openshift * Memory Limit=512Mi * Git Repository URL=https://github.com/sclorg/httpd-ex.git * Git Reference= * Context Directory= * Application Hostname= * GitHub Webhook Secret=hBTKoFTq3QQ8URHFLSMQoj3U6cArbmS3NOqIls5O # generated * Generic Webhook Secret=P8jOrOmdaXvfwNT2BboklnfTMYgjtCrf0ect8jd1 # generated --> Creating resources ... service "httpd-example" created route.route.openshift.io "httpd-example" created imagestream.image.openshift.io "httpd-example" created buildconfig.build.openshift.io "httpd-example" created deploymentconfig.apps.openshift.io "httpd-example" created --> Success Access your application via route 'httpd-example-knarra.apps.yinzhou-vs-6.qe.devcluster.openshift.com' Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress. Run 'oc status' to view your app. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 20s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 28s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 35s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 0/1 Completed 0 2m3s httpd-example-1-deploy 0/1 Completed 0 87s httpd-example-1-ks4nm 1/1 Running 0 80s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc create quota test1 --hard=openshift.io/imagestreams=10 resourcequota/test1 created [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: knarra Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc delete all -l app=httpd-example replicationcontroller "httpd-example-1" deleted service "httpd-example" deleted deploymentconfig.apps.openshift.io "httpd-example" deleted buildconfig.build.openshift.io "httpd-example" deleted imagestream.image.openshift.io "httpd-example" deleted route.route.openshift.io "httpd-example" deleted [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: knarra Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 Based on the above moving bug to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633