Bug 1879430
| Summary: | openshift-object-counts quota is not dynamically updating as the resource is deleted. | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sahil thakare <sthakare> | |
| Component: | kube-controller-manager | Assignee: | Lukasz Szaszkiewicz <lszaszki> | |
| Status: | CLOSED ERRATA | QA Contact: | RamaKasturi <knarra> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 4.4 | CC: | aos-bugs, igreen, jokerman, knarra, lszaszki, maszulik, mfojtik, sttts | |
| Target Milestone: | --- | |||
| Target Release: | 4.7.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1891518 (view as bug list) | Environment: | ||
| Last Closed: | 2021-02-24 15:18:31 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1891518 | |||
|
Description
Sahil thakare
2020-09-16 09:52:47 UTC
Not sure if this is KCM or OCM, but sending to KCM because Quota. Could also be OCM if some sort of cleanup is failing to happen on imagestreams. Seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=1874651 sending over to openshift-apiserver team Maciej, i see that a customer is being effected as per the https://bugzilla.redhat.com/show_bug.cgi?id=1874651#c5, is it possible to have a fix for this in 4.6 ? Or based on your comment 2 should we close this bug as duplicate ? (In reply to RamaKasturi from comment #3) > Maciej, i see that a customer is being effected as per the > https://bugzilla.redhat.com/show_bug.cgi?id=1874651#c5, is it possible to > have a fix for this in 4.6 ? Or based on your comment 2 should we close this > bug as duplicate ? I'm not 100% sure it's duplicate, although it seems similar I'll leave Lukasz that decision, before closing this as duplicate. Also with the current assignements, I doubt we'll be able to fix that right away. At most we might consider backporting the fix to 4.6.z, if needed. I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint. I was able to reproduce the issue today on a 4.7 cluster. I don't think it is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1874651. I'm trying to find the root cause of the issue. The quota is not dynamically updated because the resource name is incorrect. It must be of the form of count/<resource>.<group> as described in https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota I tested it successfully with the following ResourceQuota: apiVersion: v1 kind: ResourceQuota metadata: name: openshift-object-counts spec: hard: count/imagestreams.image.openshift.io: "10" I did a few manual tests to see why it went up but not down once set to "openshift.io/imagestreams" It looks like the admission plugin in the openshift-apiserver had more info to properly resolve "openshift.io/imagestreams" so it went up during the object creation. I didn't go down because KCM wasn't unable to resolve "openshift.io/imagestreams". In general "openshift.io/imagestreams" is accepted because the validation only checks the name which is valid. I'm investigating this further to see what could be done to warn the end-user earlier. Found it, the openshift-apiserver maintains an alias ("openshift.io/imagestreams") https://github.com/openshift/openshift-apiserver/blob/master/pkg/quota/image/registry.go#L16
That's why the quota was calculated on creation but wasn't' during removal. KCM doesn't know about this alias.
I'm not sure what is our depreciation policy (I'm going to check that). I can imagine it can cause some issues during migration from 3.X to 4.X.
In that case, we should update KCM so that it knows about the custom alias.
Verified bug with the payload below and i see that the fix works fine. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc version Client Version: 4.7.0-0.nightly-2020-11-10-023606 Server Version: 4.7.0-0.nightly-2020-11-10-023606 Kubernetes Version: v1.19.2+7e80e12 Below are the steps followed to verify the bug in 4.7: ======================================================= [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-project test Now using project "test" on server "https://api.knarra1110.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-app httpd-example --> Deploying template "openshift/httpd-example" to project test Apache HTTP Server --------- An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. The following service(s) have been created in your project: httpd-example. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. * With parameters: * Name=httpd-example * Namespace=openshift * Memory Limit=512Mi * Git Repository URL=https://github.com/sclorg/httpd-ex.git * Git Reference= * Context Directory= * Application Hostname= * GitHub Webhook Secret=0gil7sRyormCtAFXHeJSgYjbGWkBbPaeSQaQJVP3 # generated * Generic Webhook Secret=TGj6I2kRTEFwojU2M5eabqTPJeQNq2icpPTJW4vA # generated --> Creating resources ... service "httpd-example" created route.route.openshift.io "httpd-example" created imagestream.image.openshift.io "httpd-example" created buildconfig.build.openshift.io "httpd-example" created deploymentconfig.apps.openshift.io "httpd-example" created --> Success Access your application via route 'httpd-example-test.apps.knarra1110.qe.devcluster.openshift.com' Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress. Run 'oc status' to view your app. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 12s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 0/1 Completed 0 66s httpd-example-1-deploy 0/1 Completed 0 37s httpd-example-1-qctph 1/1 Running 0 35s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc create quota test1 --hard=openshift.io/imagestreams=10 resourcequota/test1 created [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: test Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc delete all -l app=httpd-example replicationcontroller "httpd-example-1" deleted service "httpd-example" deleted deploymentconfig.apps.openshift.io "httpd-example" deleted buildconfig.build.openshift.io "httpd-example" deleted imagestream.image.openshift.io "httpd-example" deleted route.route.openshift.io "httpd-example" deleted [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: test Resource Used Hard -------- ---- ---- openshift.io/imagestreams 0 10 Tried the same in 4.6 and i see issue is reproducible: ======================================================= [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-project knarra Now using project "knarra" on server "https://api.yinzhou-vs-6.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-app httpd-example --> Deploying template "openshift/httpd-example" to project knarra Apache HTTP Server --------- An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. The following service(s) have been created in your project: httpd-example. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md. * With parameters: * Name=httpd-example * Namespace=openshift * Memory Limit=512Mi * Git Repository URL=https://github.com/sclorg/httpd-ex.git * Git Reference= * Context Directory= * Application Hostname= * GitHub Webhook Secret=hBTKoFTq3QQ8URHFLSMQoj3U6cArbmS3NOqIls5O # generated * Generic Webhook Secret=P8jOrOmdaXvfwNT2BboklnfTMYgjtCrf0ect8jd1 # generated --> Creating resources ... service "httpd-example" created route.route.openshift.io "httpd-example" created imagestream.image.openshift.io "httpd-example" created buildconfig.build.openshift.io "httpd-example" created deploymentconfig.apps.openshift.io "httpd-example" created --> Success Access your application via route 'httpd-example-knarra.apps.yinzhou-vs-6.qe.devcluster.openshift.com' Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress. Run 'oc status' to view your app. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 20s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 28s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 1/1 Running 0 35s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods NAME READY STATUS RESTARTS AGE httpd-example-1-build 0/1 Completed 0 2m3s httpd-example-1-deploy 0/1 Completed 0 87s httpd-example-1-ks4nm 1/1 Running 0 80s [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc create quota test1 --hard=openshift.io/imagestreams=10 resourcequota/test1 created [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: knarra Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc delete all -l app=httpd-example replicationcontroller "httpd-example-1" deleted service "httpd-example" deleted deploymentconfig.apps.openshift.io "httpd-example" deleted buildconfig.build.openshift.io "httpd-example" deleted imagestream.image.openshift.io "httpd-example" deleted route.route.openshift.io "httpd-example" deleted [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1 Name: test1 Namespace: knarra Resource Used Hard -------- ---- ---- openshift.io/imagestreams 1 10 Based on the above moving bug to verified state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |