Bug 1477601 - Two secrets triggered quota in Starter tier
Two secrets triggered quota in Starter tier
Product: OpenShift Online
Classification: Red Hat
Component: Master (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jordan Liggitt
Chuan Yu
: OnlineStarter
Depends On:
  Show dependency treegraph
Reported: 2017-08-02 08:47 EDT by Devan Goodwin
Modified: 2017-08-16 10:02 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-08-16 10:02:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Devan Goodwin 2017-08-02 08:47:28 EDT
Description of problem:

On deploying an application in the Starter tier, my app was configured to use two custom secrets. The second secret triggered a quota violation and I was unable to create:

Error from server (Forbidden): secrets "secret2" is forbidden: exceeded quota: object-counts, requested: secrets=1, used: secrets=20, limited: secrets=20

However at the time I saw only these (7) secrets in my project:

$ oc get secrets
NAME                   TYPE                                  DATA      AGE
builder-token-6zp7m    kubernetes.io/service-account-token   4         25s
builder-token-xj3mp    kubernetes.io/service-account-token   0         25s
default-token-40m18    kubernetes.io/service-account-token   0         25s
default-token-hm93d    kubernetes.io/service-account-token   4         25s
deployer-token-kk97n   kubernetes.io/service-account-token   4         25s
deployer-token-ncqxb   kubernetes.io/service-account-token   0         25s
sshsecret              kubernetes.io/ssh-auth                1         21s

Even more strangely, in the web console my quota page showed 12 of 20 used.

Version-Release number of selected component (if applicable):

$ oc version
oc v3.6.0-alpha.2+3c221d5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://api.starter-us-east-1.openshift.com:443
openshift v3.6.170
kubernetes v1.6.1+5115d708d7

How reproducible:

May be difficult to reproduce, at the time I could reproduce it, today I can create additional secrets without issue.

Steps to Reproduce:

1. Start new project.
2. Create a secret.
3. Create another secret.

Actual results:

Second custom secret was rejected for quota violation.

Expected results:

If a quota of 20 secrets, and we get 6 automatically for the default service accounts, I would expect to be able to create 14 for my project.

Additional info:
Comment 1 Jordan Liggitt 2017-08-02 13:14:05 EDT
during the initial project creation, it is possible for the automatically created secrets to be retried by the secret-creating controller.

if that occurrs, quota can temporarily indicate higher objects counts in use. on reconciliation, the in-use number reduces to match what is actually being used. that can take anywhere from a few seconds to a few minutes, depending on cluster size.

can you verify that quota indicates the correct number in use now?
Comment 2 Devan Goodwin 2017-08-16 08:49:07 EDT
Count is now correct. I also may have seen some indication it was related to deleting the project, then recreating it, if that is any help. I tend to do that when working on a template to clean everything up and re-try. In any case the issue does seem to go away at some point after it's hit, shortly after creating or recreating a project.
Comment 3 Jordan Liggitt 2017-08-16 10:02:10 EDT
This is working as design, with temporary overallocation possible and reconciliation coming behind and recalculating quota periodically

Note You need to log in before you can comment on or make changes to this bug.