Bug 1400200 - ClusterResourceQuota status total doesn't match sum of namespaces
Summary: ClusterResourceQuota status total doesn't match sum of namespaces
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.3.1
Assignee: Jordan Liggitt
QA Contact: Qixuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-30 16:56 UTC by Jordan Liggitt
Modified: 2016-12-07 20:59 UTC (History)
4 users (show)

Fixed In Version: atomic-openshift-3.3.1.7-1.git.0.0988966.el7
Doc Type: Bug Fix
Doc Text:
Previously when rapidly updating multiple namespaces controlled by a single ClusterResourceQuota, the status.total.used can get out of sync with the sum of the status.namespaces[*].used. Now the ClusterResourceQuota objects are properly updated.
Clone Of:
Environment:
Last Closed: 2016-12-07 20:59:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2915 0 normal SHIPPED_LIVE Important: atomic-openshift security and bug fix update 2016-12-08 01:58:10 UTC

Description Jordan Liggitt 2016-11-30 16:56:34 UTC
Description of problem:

(spawned from https://github.com/openshift/origin/issues/11560)

When rapidly updating multiple namespaces controlled by a single clusterresourcequota, the status.total.used can get out of sync with the sum of the status.namespaces[*].used


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

oc delete project foo bar
oc delete clusterresourcequota --all
oc create clusterquota for-user-by-annotation --project-annotation-selector=openshift.io/requester=user --hard=secrets=50
oc new-project foo --as=user
oc new-project bar --as=user
oc get clusterresourcequota/for-user-by-annotation -o yaml

Run that repeatedly until the status.total.used count is observed to not match sum(status.namespaces[*].used)


Actual results:


Expected results:


Additional info:

Comment 1 Jordan Liggitt 2016-11-30 17:06:34 UTC
fixed in origin 1.4.0 by https://github.com/openshift/origin/pull/11595/

fixed in ose >= 3.4.0.17 by https://github.com/openshift/ose/commit/c37bec075312db6c1399fbaa7a975787922f5dbe

will be fixed in origin 1.3.x by https://github.com/openshift/origin/pull/12067/

will be fixed in ose 3.3.x by https://github.com/openshift/ose/pull/484

Comment 5 Qixuan Wang 2016-12-02 08:39:58 UTC
Tested on OCP 3.3.1.7(openshift v3.3.1.7, kubernetes v1.3.0+52492b4, etcd 2.3.0+git), delete a namespace which can be counted by clusterresourcequota, then get the clusterresourcequota info, resources under that namespace release(here status.total.used.secrets decrease by 9) and can be used again, but a little problem is that namespace info still remains.

Here are steps
1. Create a clusterresourcequota resource with project-annotation-selector.
# oc create clusterresourcequota crq-1 --project-annotation-selector=openshift.io/requester=user --hard=secrets=50,pods=20
clusterresourcequota "crq-1" created


2. Create namespaces with annotation matched.
# oc new-project p-1 --as=user
# oc new-project p-2 --as=user
# oc get project | grep p-
p-1                                 Active
p-2                                 Active
# oc get clusterresourcequota crq-1 -o json
{
    "kind": "ClusterResourceQuota",
    "apiVersion": "v1",
    "metadata": {
        "name": "crq-1",
        "selfLink": "/oapi/v1/clusterresourcequotas/crq-1",
        "uid": "3a9505a7-b85f-11e6-b258-0ac1389dbd42",
        "resourceVersion": "67556",
        "creationTimestamp": "2016-12-02T07:16:22Z"
    },
    "spec": {
        "selector": {
            "labels": null,
            "annotations": {
                "openshift.io/requester": "user"
            }
        },
        "quota": {
            "hard": {
                "pods": "20",
                "secrets": "50"
            }
        }
    },
    "status": {
        "total": {
            "hard": {
                "pods": "20",
                "secrets": "50"
            },
            "used": {
                "pods": "0",
                "secrets": "18"
            }
        },
        "namespaces": [
            {
                "namespace": "p-1",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "9"
                    }
                }
            },
            {
                "namespace": "p-2",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "9"
                    }
                }
            }
        ]
    }
}


3. Delete a namespace.
# oc delete project p-1
# oc get project | grep p-
p-2                                 Active


4. Check clusterresourcequota info, p-1 namespace is there.
# oc get clusterresourcequota crq-1 -o json
{
    "kind": "ClusterResourceQuota",
    "apiVersion": "v1",
    "metadata": {
        "name": "crq-1",
        "selfLink": "/oapi/v1/clusterresourcequotas/crq-1",
        "uid": "3a9505a7-b85f-11e6-b258-0ac1389dbd42",
        "resourceVersion": "68388",
        "creationTimestamp": "2016-12-02T07:16:22Z"
    },
    "spec": {
        "selector": {
            "labels": null,
            "annotations": {
                "openshift.io/requester": "user"
            }
        },
        "quota": {
            "hard": {
                "pods": "20",
                "secrets": "50"
            }
        }
    },
    "status": {
        "total": {
            "hard": {
                "pods": "20",
                "secrets": "50"
            },
            "used": {
                "pods": "0",
                "secrets": "9"
            }
        },
        "namespaces": [
            {
                "namespace": "p-1",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "0"
                    }
                }
            },
            {
                "namespace": "p-2",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "9"
                    }
                }
            }
        ]
    }
}


5. Create a namespace with annotation matched again.
# oc new-project p-4 --as=user
# oc get project | grep p-
p-2                               Active
p-4                               Active
# oc get clusterresourcequota crq-1 -o json
{
    "kind": "ClusterResourceQuota",
    "apiVersion": "v1",
    "metadata": {
        "name": "crq-1",
        "selfLink": "/oapi/v1/clusterresourcequotas/crq-1",
        "uid": "3a9505a7-b85f-11e6-b258-0ac1389dbd42",
        "resourceVersion": "75173",
        "creationTimestamp": "2016-12-02T07:16:22Z"
    },
    "spec": {
        "selector": {
            "labels": null,
            "annotations": {
                "openshift.io/requester": "user"
            }
        },
        "quota": {
            "hard": {
                "pods": "20",
                "secrets": "50"
            }
        }
    },
    "status": {
        "total": {
            "hard": {
                "pods": "20",
                "secrets": "50"
            },
            "used": {
                "pods": "0",
                "secrets": "18"
            }
        },
        "namespaces": [
            {
                "namespace": "p-1",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "0"
                    }
                }
            },
            {
                "namespace": "p-2",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "9"
                    }
                }
            },
            {
                "namespace": "p-4",
                "status": {
                    "hard": {
                        "pods": "20",
                        "secrets": "50"
                    },
                    "used": {
                        "pods": "0",
                        "secrets": "9"
                    }
                }
            }
        ]
    }
}

Comment 6 Jordan Liggitt 2016-12-02 17:42:39 UTC
The deleted namespace not being removed is a separate issue, and will require a separate fix (in 3.3.x and 3.4.x). Opened https://bugzilla.redhat.com/show_bug.cgi?id=1401073 to track that

This bug is a fix for the sum(status.namespaces.used) != total.used

Comment 8 errata-xmlrpc 2016-12-07 20:59:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:2915


Note You need to log in before you can comment on or make changes to this bug.