Bug 1484134 - [RFE] Objects should pass validation prior to charging quota
Summary: [RFE] Objects should pass validation prior to charging quota
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RFE
Version: 3.4.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Eric Paris
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1485375
TreeView+ depends on / blocked
 
Reported: 2017-08-22 19:11 UTC by Freddy E. Montero
Modified: 2019-02-11 18:14 UTC (History)
9 users (show)

(edit)
Clone Of:
: 1485375 (view as bug list)
(edit)
Last Closed:


Attachments (Terms of Use)

Description Freddy E. Montero 2017-08-22 19:11:09 UTC
Description of problem:
If the wrong secret is passed in a given buildconfig (secret is non existing in the project space), it will consume all of the resources set via quota and thus any other build, deployment, etc. will fail until the X build is deleted.

Version-Release number of selected component (if applicable):
3.4.1.44

How reproducible:
To reproduce this scenario, we used an invalid source secret in a BuildConfig:
    sourceSecret:
      name: HUBXTFS
then we started the build.
This produced the following event logs:

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   1         hubx-hfcs-bayassignmentupdate-25   Build                                  Warning   HandleBuildError   {build-controller }                              Build has error: failed to create build pod: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:32:23 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   42        hubx-hfcs-bayassignmentupdate-25   Build                                  Warning   FailedCreate       {build-controller }                              Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:32:24 -0400 EDT   2017-08-22 14:32:24 -0400 EDT   1         hubx-hfcs-bayassignmentupdate-25   Build                                  Warning   HandleBuildError   {build-controller }                              Build has error: failed to create build pod: pods "hubx-hfcs-bayassignmentupdate-25-build" is forbidden: exceeded quota: compute, requested: limits.memory=1600Mi, used: limits.memory=81700Mi, limited: limits.memory=80Gi

2017-08-22 14:33:38 -0400 EDT   2017-08-22 14:32:24 -0400 EDT   75        hubx-hfcs-bayassignmentupdate-25   Build                                  Warning   FailedCreate       {build-controller }                              Error creating: pods "hubx-hfcs-bayassignmentupdate-25-build" is forbidden: exceeded quota: compute, requested: limits.memory=1600Mi, used: limits.memory=81700Mi, limited: limits.memory=80Gi

This will consume all cpu/memory from the quota, even though we have limits set
[root@~]# oc describe limits compute -n hubx-build
Name:           compute
Namespace:      hubx-build
Type            Resource        Min     Max     Default Request Default Limit   Max Limit/Request Ratio
----            --------        ---     ---     --------------- -------------   -----------------------
Pod             cpu             50m     8       -               -               -
Pod             memory          4Mi     8Gi     -               -               -
Container       memory          4Mi     8Gi     1Gi             1Gi             -
Container       cpu             50m     8       50m             1               50


Expected results:
18m       18m       1         hubx-hfcs-bayassignmentupdate-29-build   Pod                                             Normal    Scheduled           {default-scheduler }                             Successfully assigned hubx-hfcs-bayassignmentupdate-29-build to njrarltapp0019e.linux.us.ams1907.com

16m       16m       1         hubx-hfcs-bayassignmentupdate-29-build   Pod                     spec.containers{sti-build}                 Normal    Pulled              {kubelet njrarltapp0019e.linux.us.ams1907.com}   Container image "nexusrmps.njrar.us.ups.com:5000/openshift3/ose-sti-builder:v3.4.1.44" already present on machine

16m       16m       1         hubx-hfcs-bayassignmentupdate-29-build   Pod                     spec.containers{sti-build}                 Normal    Created             {kubelet njrarltapp0019e.linux.us.ams1907.com}   Created container with docker id 0361fc8056af; Security:[seccomp=unconfined]

16m       16m       1         hubx-hfcs-bayassignmentupdate-29-build   Pod                     spec.containers{sti-build}                 Normal    Started             {kubelet njrarltapp0019e.linux.us.ams1907.com}   Started container with docker id 0361fc8056af

Comment 1 Ben Parees 2017-08-22 19:32:41 UTC
we have a separate issue(https://github.com/openshift/origin/issues/15876) open to address how the invalid mount name should be fixed when we generate a mount name from a secret, but failing to create pods should not consume the user's quota, so assigning to k8s component for triage on that front.

Freddy, are the build pods actually being created successfully?  i.e. do you see multiple build pods created for the build?

Comment 2 Freddy E. Montero 2017-08-22 19:53:18 UTC
(In reply to Ben Parees from comment #1)
> we have a separate issue(https://github.com/openshift/origin/issues/15876)
> open to address how the invalid mount name should be fixed when we generate
> a mount name from a secret, but failing to create pods should not consume
> the user's quota, so assigning to k8s component for triage on that front.
> 
> Freddy, are the build pods actually being created successfully?  i.e. do you
> see multiple build pods created for the build?

When we fixed the secret in the BuildConfig then we can create pods successfully.

If the BC is not fixed and application developers pushes a change, the build will be stuck/fail.
 
hubx-hfcs-bayassignmentupdate-20   Source    Git@dev       Failed (ExceededRetryTimeout)
hubx-hfcs-bayassignmentupdate-21   Source    Git@dev       Failed (ExceededRetryTimeout)
hubx-hfcs-bayassignmentupdate-28   Source    Git@cc49808   Failed                          About an hour ago   25s
hubx-hfcs-bayassignmentupdate-29   Source    Git@956de4f   Failed                          59 minutes ago      27s

Comment 3 Ben Parees 2017-08-22 20:02:23 UTC
ok, I just wanted to confirm that no pods were actually being created when you were in the broken state, which makes it strange that k8s would thing you have consumed your quota.

Comment 4 Freddy E. Montero 2017-08-22 20:11:14 UTC
(In reply to Ben Parees from comment #3)
> ok, I just wanted to confirm that no pods were actually being created when
> you were in the broken state, which makes it strange that k8s would thing
> you have consumed your quota.

What I noticed and forgot to included was that this specific error runs until all quota is consumed.
See below:

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   1         hubx-hfcs-bayassignmentupdate-25   Build                 Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   1         hubx-hfcs-bayassignmentupdate-25   Build               Warning   HandleBuildError   {build-controller }   Build has error: failed to create build pod: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   2         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   3         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), 
spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   4         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   5         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   6         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   7         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:52 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   8         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:53 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   9         hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:53 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   10        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:53 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   11        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:53 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   12        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:54 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   13        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:55 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   14        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:56 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   15        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:57 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   16        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:58 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   17        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:31:59 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   18        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:32:00 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   19        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:32:01 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   20        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:32:02 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   21        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

2017-08-22 14:32:03 -0400 EDT   2017-08-22 14:31:52 -0400 EDT   22        hubx-hfcs-bayassignmentupdate-25   Build               Warning   FailedCreate   {build-controller }   Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]

Comment 5 Freddy E. Montero 2017-08-22 20:19:38 UTC
(In reply to Ben Parees from comment #1)
> we have a separate issue(https://github.com/openshift/origin/issues/15876)
> open to address how the invalid mount name should be fixed when we generate
> a mount name from a secret, but failing to create pods should not consume
> the user's quota, so assigning to k8s component for triage on that front.
> 
> Freddy, are the build pods actually being created successfully?  i.e. do you
> see multiple build pods created for the build?

Ben,
It seems besides the issue with . in the secret name, using UPPPERCASES also causes the regex to fail.

Comment 6 Ben Parees 2017-08-22 20:37:17 UTC
Right, I just wanted to make sure somehow the pod objects weren't getting created in spite of the claimed validation error.  I don't understand what has consumed your quota, given that the pods are not being created, so that's why I think this (the quota issue) is a k8s bug.  

The pods failing to be created is a build bug which we'll track in the aforementioned github issue.

Comment 7 Derek Carr 2017-08-23 15:51:19 UTC
quota is incremented in admission prior to validation of the resource.  as a result, quota may be incremented even if the pod is not ultimately persisted.  this is a known limitation of the quota subsystem today in kubernetes that we will have to address in a future release.

we should probably doc this at minimum in the product documentation, and we would want an RFE in the future to address this.

Comment 8 Eric Rich 2017-08-24 15:11:49 UTC
(In reply to Derek Carr from comment #7)
> quota is incremented in admission prior to validation of the resource.  as a
> result, quota may be incremented even if the pod is not ultimately
> persisted.  this is a known limitation of the quota subsystem today in
> kubernetes that we will have to address in a future release.
> 
> we should probably doc this at minimum in the product documentation, and we
> would want an RFE in the future to address this.

If this is true, what the mitigation for a customer facing an issue where a failed build / pod, simply continues to consume the projects quota? 

It seems the only answer is to delete the build, which may take time to notice.

Comment 9 Derek Carr 2017-08-24 21:29:46 UTC
To provide context for this issue, the API server at a high level works as follows:

1. Receive request
2. Deserialize the object
3. Default the object
4. Convert the object to internal form
5. Admission controllers Admit the object
6. Validate the object
7. ...
8. Persist the Object

Quota validation happens as part of the admission controller chain, and before object validation.  As a result, invalid objects sent to the API server can cause temporary charges to quota until replenishment occurs.

Ideally, we could have validation happen before quota, which requires bucketing of admission controllers into defaulters and non-defaulters.  This bucketing is not yet possible.

In the interim, we have made some special allowances for pods to do some validation of the resource in the quota admission chain.

See as an example:
https://github.com/kubernetes/kubernetes/pull/25487

We should try to get this fixed in Kubernetes 1.8 to expand to the full pod spec, and backport to 3.7 release.

Comment 11 Michal Fojtik 2017-09-14 11:37:09 UTC
(In reply to Derek Carr from comment #10)
> Upstream PR to validate pod before quota charging:
> https://github.com/kubernetes/kubernetes/pull/51370

Will that PR land in 3.7? Or we need to bump the target release?

Comment 12 Derek Carr 2017-09-15 16:46:47 UTC
Upstream PR not yet merged.

Comment 14 Derek Carr 2017-10-17 19:12:17 UTC
there is work proposed in the upstream to split admission control into a mutating and validation phase.

the design is tracked here:
https://github.com/kubernetes/community/pull/1137#discussion_r142964564

Comment 15 Will Gordon 2018-10-30 15:19:37 UTC
Hey all,

This bug has seemed to go nowhere. I have an OpenShift Online customer who has just hit this, and the extra oddity here is there is a ClusterResourceQuota and a ProjectQuota.

When the invalid build uses all of the quota, it ends up using all of the ClusterResourceQuota and none of the ProjectQuota. So it effectively prevents any builds in any owned namespaces.

I would have expected the ProjectQuota to supersede the ClusterResourceQuota first. Is this the same issue, or should this be tracked in a new BZ?


Note You need to log in before you can comment on or make changes to this bug.