Description of problem: If the wrong secret is passed in a given buildconfig (secret is non existing in the project space), it will consume all of the resources set via quota and thus any other build, deployment, etc. will fail until the X build is deleted. Version-Release number of selected component (if applicable): 3.4.1.44 How reproducible: To reproduce this scenario, we used an invalid source secret in a BuildConfig: sourceSecret: name: HUBXTFS then we started the build. This produced the following event logs: 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 1 hubx-hfcs-bayassignmentupdate-25 Build Warning HandleBuildError {build-controller } Build has error: failed to create build pod: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:32:23 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 42 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:32:24 -0400 EDT 2017-08-22 14:32:24 -0400 EDT 1 hubx-hfcs-bayassignmentupdate-25 Build Warning HandleBuildError {build-controller } Build has error: failed to create build pod: pods "hubx-hfcs-bayassignmentupdate-25-build" is forbidden: exceeded quota: compute, requested: limits.memory=1600Mi, used: limits.memory=81700Mi, limited: limits.memory=80Gi 2017-08-22 14:33:38 -0400 EDT 2017-08-22 14:32:24 -0400 EDT 75 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: pods "hubx-hfcs-bayassignmentupdate-25-build" is forbidden: exceeded quota: compute, requested: limits.memory=1600Mi, used: limits.memory=81700Mi, limited: limits.memory=80Gi This will consume all cpu/memory from the quota, even though we have limits set [root@~]# oc describe limits compute -n hubx-build Name: compute Namespace: hubx-build Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Pod cpu 50m 8 - - - Pod memory 4Mi 8Gi - - - Container memory 4Mi 8Gi 1Gi 1Gi - Container cpu 50m 8 50m 1 50 Expected results: 18m 18m 1 hubx-hfcs-bayassignmentupdate-29-build Pod Normal Scheduled {default-scheduler } Successfully assigned hubx-hfcs-bayassignmentupdate-29-build to njrarltapp0019e.linux.us.ams1907.com 16m 16m 1 hubx-hfcs-bayassignmentupdate-29-build Pod spec.containers{sti-build} Normal Pulled {kubelet njrarltapp0019e.linux.us.ams1907.com} Container image "nexusrmps.njrar.us.ups.com:5000/openshift3/ose-sti-builder:v3.4.1.44" already present on machine 16m 16m 1 hubx-hfcs-bayassignmentupdate-29-build Pod spec.containers{sti-build} Normal Created {kubelet njrarltapp0019e.linux.us.ams1907.com} Created container with docker id 0361fc8056af; Security:[seccomp=unconfined] 16m 16m 1 hubx-hfcs-bayassignmentupdate-29-build Pod spec.containers{sti-build} Normal Started {kubelet njrarltapp0019e.linux.us.ams1907.com} Started container with docker id 0361fc8056af
we have a separate issue(https://github.com/openshift/origin/issues/15876) open to address how the invalid mount name should be fixed when we generate a mount name from a secret, but failing to create pods should not consume the user's quota, so assigning to k8s component for triage on that front. Freddy, are the build pods actually being created successfully? i.e. do you see multiple build pods created for the build?
(In reply to Ben Parees from comment #1) > we have a separate issue(https://github.com/openshift/origin/issues/15876) > open to address how the invalid mount name should be fixed when we generate > a mount name from a secret, but failing to create pods should not consume > the user's quota, so assigning to k8s component for triage on that front. > > Freddy, are the build pods actually being created successfully? i.e. do you > see multiple build pods created for the build? When we fixed the secret in the BuildConfig then we can create pods successfully. If the BC is not fixed and application developers pushes a change, the build will be stuck/fail. hubx-hfcs-bayassignmentupdate-20 Source Git@dev Failed (ExceededRetryTimeout) hubx-hfcs-bayassignmentupdate-21 Source Git@dev Failed (ExceededRetryTimeout) hubx-hfcs-bayassignmentupdate-28 Source Git@cc49808 Failed About an hour ago 25s hubx-hfcs-bayassignmentupdate-29 Source Git@956de4f Failed 59 minutes ago 27s
ok, I just wanted to confirm that no pods were actually being created when you were in the broken state, which makes it strange that k8s would thing you have consumed your quota.
(In reply to Ben Parees from comment #3) > ok, I just wanted to confirm that no pods were actually being created when > you were in the broken state, which makes it strange that k8s would thing > you have consumed your quota. What I noticed and forgot to included was that this specific error runs until all quota is consumed. See below: 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 1 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 1 hubx-hfcs-bayassignmentupdate-25 Build Warning HandleBuildError {build-controller } Build has error: failed to create build pod: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 2 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 3 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 4 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 5 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 6 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 7 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:52 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 8 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:53 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 9 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:53 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 10 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:53 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 11 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:53 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 12 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:54 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 13 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:55 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 14 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:56 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 15 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:57 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 16 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:58 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 17 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:31:59 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 18 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:32:00 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 19 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:32:01 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 20 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:32:02 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 21 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"] 2017-08-22 14:32:03 -0400 EDT 2017-08-22 14:31:52 -0400 EDT 22 hubx-hfcs-bayassignmentupdate-25 Build Warning FailedCreate {build-controller } Error creating: Pod "hubx-hfcs-bayassignmentupdate-25-build" is invalid: [spec.volumes[3].name: Invalid value: "HUBXTFS-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[3].name: Not found: "HUBXTFS-source"]
(In reply to Ben Parees from comment #1) > we have a separate issue(https://github.com/openshift/origin/issues/15876) > open to address how the invalid mount name should be fixed when we generate > a mount name from a secret, but failing to create pods should not consume > the user's quota, so assigning to k8s component for triage on that front. > > Freddy, are the build pods actually being created successfully? i.e. do you > see multiple build pods created for the build? Ben, It seems besides the issue with . in the secret name, using UPPPERCASES also causes the regex to fail.
Right, I just wanted to make sure somehow the pod objects weren't getting created in spite of the claimed validation error. I don't understand what has consumed your quota, given that the pods are not being created, so that's why I think this (the quota issue) is a k8s bug. The pods failing to be created is a build bug which we'll track in the aforementioned github issue.
quota is incremented in admission prior to validation of the resource. as a result, quota may be incremented even if the pod is not ultimately persisted. this is a known limitation of the quota subsystem today in kubernetes that we will have to address in a future release. we should probably doc this at minimum in the product documentation, and we would want an RFE in the future to address this.
(In reply to Derek Carr from comment #7) > quota is incremented in admission prior to validation of the resource. as a > result, quota may be incremented even if the pod is not ultimately > persisted. this is a known limitation of the quota subsystem today in > kubernetes that we will have to address in a future release. > > we should probably doc this at minimum in the product documentation, and we > would want an RFE in the future to address this. If this is true, what the mitigation for a customer facing an issue where a failed build / pod, simply continues to consume the projects quota? It seems the only answer is to delete the build, which may take time to notice.
To provide context for this issue, the API server at a high level works as follows: 1. Receive request 2. Deserialize the object 3. Default the object 4. Convert the object to internal form 5. Admission controllers Admit the object 6. Validate the object 7. ... 8. Persist the Object Quota validation happens as part of the admission controller chain, and before object validation. As a result, invalid objects sent to the API server can cause temporary charges to quota until replenishment occurs. Ideally, we could have validation happen before quota, which requires bucketing of admission controllers into defaulters and non-defaulters. This bucketing is not yet possible. In the interim, we have made some special allowances for pods to do some validation of the resource in the quota admission chain. See as an example: https://github.com/kubernetes/kubernetes/pull/25487 We should try to get this fixed in Kubernetes 1.8 to expand to the full pod spec, and backport to 3.7 release.
(In reply to Derek Carr from comment #10) > Upstream PR to validate pod before quota charging: > https://github.com/kubernetes/kubernetes/pull/51370 Will that PR land in 3.7? Or we need to bump the target release?
Upstream PR not yet merged.
there is work proposed in the upstream to split admission control into a mutating and validation phase. the design is tracked here: https://github.com/kubernetes/community/pull/1137#discussion_r142964564
Hey all, This bug has seemed to go nowhere. I have an OpenShift Online customer who has just hit this, and the extra oddity here is there is a ClusterResourceQuota and a ProjectQuota. When the invalid build uses all of the quota, it ends up using all of the ClusterResourceQuota and none of the ProjectQuota. So it effectively prevents any builds in any owned namespaces. I would have expected the ProjectQuota to supersede the ClusterResourceQuota first. Is this the same issue, or should this be tracked in a new BZ?
With the introduction of OpenShift 4, Red Hat has delivered or roadmapped a substantial number of features based on feedback by our customers. Many of the enhancements encompass specific RFEs which have been requested, or deliver a comparable solution to a customer problem, rendering an RFE redundant. This bz (RFE) has been identified as a feature request not yet planned or scheduled for an OpenShift release and is being closed. If this feature is still an active request that needs to be tracked, Red Hat Support can assist in filing a request in the new JIRA RFE system, as well as provide you with updates as the RFE progress within our planning processes. Please open a new support case: https://access.redhat.com/support/cases/#/case/new Opening a New Support Case: https://access.redhat.com/support/cases/#/case/new As the new Jira RFE system is not yet public, Red Hat Support can help answer your questions about your RFEs via the same support case system.