Bug 1879430 - openshift-object-counts quota is not dynamically updating as the resource is deleted.
Summary: openshift-object-counts quota is not dynamically updating as the resource is ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Lukasz Szaszkiewicz
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On:
Blocks: 1891518
TreeView+ depends on / blocked
 
Reported: 2020-09-16 09:52 UTC by Sahil thakare
Modified: 2021-02-24 15:19 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1891518 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:18:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-policy-controller pull 44 0 None closed runs OpenShift specific ResourceQuota monitors 2021-01-09 21:22:07 UTC
Github openshift kubernetes pull 424 0 None closed UPSTREAM <carry>: maintains OpenShift specific ResourceQuota evaluators 2021-01-09 21:22:44 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:19:03 UTC

Description Sahil thakare 2020-09-16 09:52:47 UTC
Description of problem:
The resource quota statistics do not update dynamically even when the resource is deleted.

Please see below tests:

On OCP 4 cluster:
~~~~~~~~~~~~~~~~~~~~~~~~~
[quicklab@upi-0 ~]$ oc version 
Client Version: 4.4.18
Server Version: 4.4.18
Kubernetes Version: v1.17.1+20ba474
~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~
[quicklab@upi-0 ~]$ oc new-app httpd-example
warning: Cannot find git. Ensure that it is installed and in your path. Git is required to work with git repositories.
--> Deploying template "openshift/httpd-example" to project test11
~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~
[quicklab@upi-0 ~]$ oc describe  quota openshift-object-counts
Name:                      openshift-object-counts
Namespace:                 test11
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  1     10
~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~
[quicklab@upi-0 ~]$ oc delete all -l app=httpd-example
replicationcontroller "httpd-example-1" deleted
service "httpd-example" deleted
deploymentconfig.apps.openshift.io "httpd-example" deleted
~~~~~~~~~~~~~~~~~~~~~~~~~~~

After an hour:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[quicklab@upi-0 ~]$ oc describe  quota openshift-object-counts
Name:                      openshift-object-counts
Namespace:                 test11
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  1     10
You have new mail in /var/spool/mail/quicklab
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However, it's working in 3.11:

1. On OCP 3.X cluster
~~~~~~~~~~~~~~~~~~~~
[quicklab@master-0 ~]$ oc version 
Server https://openshift.internal.sthakare11.lab.pnq2.cee.redhat.com:443
openshift v3.11.272
kubernetes v1.11.0+d4cacc0
~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~
[quicklab@master-0 ~]$ oc new-app httpd-example
--> Deploying template "openshift/httpd-example" to project test12
~~~~~~~~~~~~~~

~~~~~~~~~~~~~~
[quicklab@master-0 ~]$ oc describe  quota openshift-object-counts
Name:                      openshift-object-counts
Namespace:                 test12
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  1     10
~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~
[quicklab@master-0 ~]$ oc delete all -l app=httpd-example
replicationcontroller "httpd-example-1" deleted
service "httpd-example" deleted
~~~~~~~~~~~~~~~~

After some time:

~~~~~~~~~~~~~~~~
<pre>[quicklab@master-0 ~]$ oc describe  quota openshift-object-counts
Name:                      openshift-object-counts
Namespace:                 test12
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  0     10
</pre>
~~~~~~~~~~~~~~~~~


Version-Release number of selected component (if applicable):

$ oc version
Client Version: 4.4.19
Server Version: 4.4.19
Kubernetes Version: v1.17.1+6af3663


How reproducible:

1. Create a  test project and application
2. Create a resource quota for the count statistics. doc[1]
3. Check the count statistics.
4. Delete the application and check the count statistics again


Steps to Reproduce:

1. Create a  test project and application
2. Create a resource quota for the count statistics. doc[1]
3. Check the count statistics.
4. Delete the application and check the count statistics again

Actual results:

Quota count statistics not updating after deleting the image stream.

Expected results:
Quota count statistics should reflect the changes dynamically.

Additional info:

[1] https://docs.openshift.com/container-platform/3.11/dev_guide/compute_resources.html#dev-viewing-quotas

Comment 1 Seth Jennings 2020-09-16 16:33:09 UTC
Not sure if this is KCM or OCM, but sending to KCM because Quota.  Could also be OCM if some sort of cleanup is failing to happen on imagestreams.

Comment 2 Maciej Szulik 2020-09-17 10:09:03 UTC
Seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=1874651 sending over to openshift-apiserver team

Comment 3 RamaKasturi 2020-09-21 16:50:43 UTC
Maciej, i see that a customer is being effected as per the https://bugzilla.redhat.com/show_bug.cgi?id=1874651#c5, is it possible to have a fix for this in 4.6 ? Or based on your comment 2 should we close this bug as duplicate ?

Comment 4 Maciej Szulik 2020-09-22 09:49:44 UTC
(In reply to RamaKasturi from comment #3)
> Maciej, i see that a customer is being effected as per the
> https://bugzilla.redhat.com/show_bug.cgi?id=1874651#c5, is it possible to
> have a fix for this in 4.6 ? Or based on your comment 2 should we close this
> bug as duplicate ?

I'm not 100% sure it's duplicate, although it seems similar I'll leave Lukasz that decision, before closing this as duplicate. 
Also with the current assignements, I doubt we'll be able to fix that right away. At most we might consider backporting
the fix to 4.6.z, if needed.

Comment 6 Lukasz Szaszkiewicz 2020-10-02 10:32:07 UTC
Iā€™m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.

Comment 7 Lukasz Szaszkiewicz 2020-10-21 14:46:22 UTC
I was able to reproduce the issue today on a 4.7 cluster. I don't think it is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1874651.
I'm trying to find the root cause of the issue.

Comment 8 Lukasz Szaszkiewicz 2020-10-22 12:51:31 UTC
The quota is not dynamically updated because the resource name is incorrect.
It must be of the form of count/<resource>.<group> as described in https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota


I tested it successfully with the following ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: openshift-object-counts
spec:
  hard:
    count/imagestreams.image.openshift.io: "10"

I did a few manual tests to see why it went up but not down once set to "openshift.io/imagestreams"
It looks like the admission plugin in the openshift-apiserver had more info to properly resolve "openshift.io/imagestreams" so it went up during the object creation.
I didn't go down because KCM wasn't unable to resolve "openshift.io/imagestreams".


In general "openshift.io/imagestreams" is accepted because the validation only checks the name which is valid.
I'm investigating this further to see what could be done to warn the end-user earlier.

Comment 9 Lukasz Szaszkiewicz 2020-10-22 14:22:29 UTC
Found it, the openshift-apiserver maintains an alias ("openshift.io/imagestreams") https://github.com/openshift/openshift-apiserver/blob/master/pkg/quota/image/registry.go#L16
That's why the quota was calculated on creation but wasn't' during removal. KCM doesn't know about this alias.

I'm not sure what is our depreciation policy (I'm going to check that). I can imagine it can cause some issues during migration from 3.X to 4.X.
In that case, we should update KCM so that it knows about the custom alias.

Comment 11 RamaKasturi 2020-11-10 08:39:20 UTC
Verified bug with the payload below and i see that the fix works fine.

[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc version
Client Version: 4.7.0-0.nightly-2020-11-10-023606
Server Version: 4.7.0-0.nightly-2020-11-10-023606
Kubernetes Version: v1.19.2+7e80e12

Below are the steps followed to verify the bug in 4.7:
=======================================================
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-project test
Now using project "test" on server "https://api.knarra1110.qe.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-app httpd-example
--> Deploying template "openshift/httpd-example" to project test

     Apache HTTP Server
     ---------
     An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md.

     The following service(s) have been created in your project: httpd-example.
     
     For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md.

     * With parameters:
        * Name=httpd-example
        * Namespace=openshift
        * Memory Limit=512Mi
        * Git Repository URL=https://github.com/sclorg/httpd-ex.git
        * Git Reference=
        * Context Directory=
        * Application Hostname=
        * GitHub Webhook Secret=0gil7sRyormCtAFXHeJSgYjbGWkBbPaeSQaQJVP3 # generated
        * Generic Webhook Secret=TGj6I2kRTEFwojU2M5eabqTPJeQNq2icpPTJW4vA # generated

--> Creating resources ...
    service "httpd-example" created
    route.route.openshift.io "httpd-example" created
    imagestream.image.openshift.io "httpd-example" created
    buildconfig.build.openshift.io "httpd-example" created
    deploymentconfig.apps.openshift.io "httpd-example" created
--> Success
    Access your application via route 'httpd-example-test.apps.knarra1110.qe.devcluster.openshift.com' 
    Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress.
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods
NAME                    READY   STATUS    RESTARTS   AGE
httpd-example-1-build   1/1     Running   0          12s
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods
NAME                     READY   STATUS      RESTARTS   AGE
httpd-example-1-build    0/1     Completed   0          66s
httpd-example-1-deploy   0/1     Completed   0          37s
httpd-example-1-qctph    1/1     Running     0          35s

[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc create quota test1 --hard=openshift.io/imagestreams=10
resourcequota/test1 created

[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1
Name:                      test1
Namespace:                 test
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  1     10

[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc delete all -l app=httpd-example
replicationcontroller "httpd-example-1" deleted
service "httpd-example" deleted
deploymentconfig.apps.openshift.io "httpd-example" deleted
buildconfig.build.openshift.io "httpd-example" deleted
imagestream.image.openshift.io "httpd-example" deleted
route.route.openshift.io "httpd-example" deleted
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1
Name:                      test1
Namespace:                 test
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  0     10


Tried the same in 4.6 and i see issue is reproducible:
=======================================================
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-project knarra
Now using project "knarra" on server "https://api.yinzhou-vs-6.qe.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc new-app httpd-example
--> Deploying template "openshift/httpd-example" to project knarra

     Apache HTTP Server
     ---------
     An example Apache HTTP Server (httpd) application that serves static content. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md.

     The following service(s) have been created in your project: httpd-example.
     
     For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/httpd-ex/blob/master/README.md.

     * With parameters:
        * Name=httpd-example
        * Namespace=openshift
        * Memory Limit=512Mi
        * Git Repository URL=https://github.com/sclorg/httpd-ex.git
        * Git Reference=
        * Context Directory=
        * Application Hostname=
        * GitHub Webhook Secret=hBTKoFTq3QQ8URHFLSMQoj3U6cArbmS3NOqIls5O # generated
        * Generic Webhook Secret=P8jOrOmdaXvfwNT2BboklnfTMYgjtCrf0ect8jd1 # generated

--> Creating resources ...
    service "httpd-example" created
    route.route.openshift.io "httpd-example" created
    imagestream.image.openshift.io "httpd-example" created
    buildconfig.build.openshift.io "httpd-example" created
    deploymentconfig.apps.openshift.io "httpd-example" created
--> Success
    Access your application via route 'httpd-example-knarra.apps.yinzhou-vs-6.qe.devcluster.openshift.com' 
    Build scheduled, use 'oc logs -f buildconfig/httpd-example' to track its progress.
    Run 'oc status' to view your app.
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods
NAME                    READY   STATUS    RESTARTS   AGE
httpd-example-1-build   1/1     Running   0          20s
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods
NAME                    READY   STATUS    RESTARTS   AGE
httpd-example-1-build   1/1     Running   0          28s
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods
NAME                    READY   STATUS    RESTARTS   AGE
httpd-example-1-build   1/1     Running   0          35s
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc get pods
NAME                     READY   STATUS      RESTARTS   AGE
httpd-example-1-build    0/1     Completed   0          2m3s
httpd-example-1-deploy   0/1     Completed   0          87s
httpd-example-1-ks4nm    1/1     Running     0          80s
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc create quota test1 --hard=openshift.io/imagestreams=10
resourcequota/test1 created
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1
Name:                      test1
Namespace:                 knarra
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  1     10
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc delete all -l app=httpd-example
replicationcontroller "httpd-example-1" deleted
service "httpd-example" deleted
deploymentconfig.apps.openshift.io "httpd-example" deleted
buildconfig.build.openshift.io "httpd-example" deleted
imagestream.image.openshift.io "httpd-example" deleted
route.route.openshift.io "httpd-example" deleted
[knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-10-023606]$ ./oc describe quota test1
Name:                      test1
Namespace:                 knarra
Resource                   Used  Hard
--------                   ----  ----
openshift.io/imagestreams  1     10


Based on the above moving bug to verified state.

Comment 15 errata-xmlrpc 2021-02-24 15:18:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.