Bug 1978627 - When mount source with a long unexist name, the build keeps pending with unclear message
Summary: When mount source with a long unexist name, the build keeps pending with uncl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Corey Daley
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-02 11:00 UTC by XiuJuan Wang
Modified: 2021-10-18 17:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-18 17:38:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-apiserver pull 230 0 None open Bug 1978627: Validate secret name and configMap name 2021-07-14 16:33:20 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:38:19 UTC

Description XiuJuan Wang 2021-07-02 11:00:12 UTC
Description of problem:
When mount source with a long unexist name, the build keeps pending with unclear message

Version-Release number of selected component (if applicable):
$ oc version 
Client Version: 4.9.0-0.nightly-2021-07-02-002856
Server Version: 4.9.0-0.nightly-2021-07-01-200539
Kubernetes Version: v1.21.0-rc.0+3ed65d9

How reproducible:
always

Steps to Reproduce:
1.Create a buildconfig
2.Mount a source secret to builder container.
$oc edit bc
  strategy:
    dockerStrategy:
      from:
        kind: ImageStreamTag
        name: centos:7
        namespace: xiuwang
      volumes:
      - mounts:
        - destinationPath: /var/run/secret
        name: mysecret
        source:
          secret:
            defaultMode: 420
            secretName: testsourceddssssssssaffffffffffffffffffffdagagddagagerqgeqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq%%4443421@!dsssssssss
          type: Secret
3.Trigger a new build

Actual results:

test1-3              Docker   Dockerfile    Pending
$oc describe builds test1-3 

Events:
  Type		Reason		Age			From			Message
  ----		------		----			----			-------
  Normal	Scheduled	4m22s			default-scheduler	Successfully assigned xiuwang/test1-3-build to ip-10-0-60-94.us-east-2.compute.internal
  Warning	FailedMount	2m20s			kubelet			Unable to attach or mount volumes: unmounted volumes=[testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume], unattached volumes=[build-proxy-ca-bundles builder-dockercfg-j65dd-push testsource-user-build-volume buildworkdir buildcachedir container-storage-root testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume build-system-configs build-ca-bundles kube-api-access-9dzxc builder-dockercfg-j65dd-pull build-blob-cache node-pullsecrets]: timed out waiting for the condition
  Warning	FailedMount	5s			kubelet			Unable to attach or mount volumes: unmounted volumes=[testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume], unattached volumes=[build-system-configs build-ca-bundles kube-api-access-9dzxc builder-dockercfg-j65dd-push testsource-user-build-volume build-proxy-ca-bundles node-pullsecrets buildworkdir build-blob-cache container-storage-root buildcachedir builder-dockercfg-j65dd-pull testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume]: timed out waiting for the condition
  Warning	FailedMount	3s (x10 over 4m22s)	kubelet			MountVolume.SetUp failed for volume "testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume" : failed to sync secret cache: timed out waiting for the condition

Expected results:

Shouldn't save for such long name or contain special string.
Additional info:

Comment 1 Gabe Montero 2021-07-06 13:05:39 UTC
Feels like there is some synergy / commonality between this BZ and the customer support case https://bugzilla.redhat.com/show_bug.cgi?id=1977414

Both deal with builds trying the mount things secrets/configmaps which are problematic.

Also perhaps some synergy / commonality with Corey's almost complete BUILD-87 feature.

Corey - by chance does you new code add immediate failure of the build if the volume is incorrect, like Adam calls for in https://bugzilla.redhat.com/show_bug.cgi?id=1977414
?

If not, seems like it should, right?

If it does, did the immediate failure you provide also apply to the existing secret/configmap source mount support?

WDYT?

Comment 2 Corey Daley 2021-07-06 14:32:38 UTC
I used existing code to mount the secrets and configmaps as volumes: https://github.com/openshift/openshift-controller-manager/pull/183/commits/356a95b7f3fc4289045a45d98374352a4f5460eb, and that code does not seem to address missing secrets/configmaps. I wonder if there is a retry buried somewhere that waits for them to appear.

Comment 3 Corey Daley 2021-07-06 14:52:23 UTC

*** This bug has been marked as a duplicate of bug 1977414 ***

Comment 4 Adam Kaplan 2021-07-13 15:59:59 UTC
Reopening this BZ - we should validate that referenced build volume names do not exceed the k8s limit on names. This is a separate issue from the pod sitting in the initializing state because the referenced ConfigMap or Secret is not present.

Comment 5 Corey Daley 2021-07-14 14:29:41 UTC
@adam.kaplan The above error message shows that the generated volume name was a truncated version of the secret name `testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume`.
`
$ irb
irb(main):001:0> "testsourceddssssssssafffffffffffffff-8176ec90-user-build-volume".length
=> 63
`

It seems like us validating the name of the secret or configmap would not make sense since if it does not adhere to the kubernetes naming requirements, than it would not exist on the cluster.

Comment 6 Adam Kaplan 2021-07-14 15:10:43 UTC
> It seems like us validating the name of the secret or configmap would not make sense since if it does not adhere to the kubernetes naming requirements, than it would not exist on the cluster.

Validating on admission provides a better user experience. If the referenced volume could never exist, we should not allow users to create the Build or BuildConfig.
If the referenced volume could exist (it does now or could be created by a user in the future), then we should allow admission.

We do this validation with referenced Secrets and ConfigMaps in the build source [1].

[1] https://github.com/openshift/openshift-apiserver/blob/master/pkg/build/apis/build/validation/validation.go#L300-L336

Comment 7 Corey Daley 2021-07-14 16:34:34 UTC
Submitted https://github.com/openshift/openshift-apiserver/pull/230 to add validation for secret name and configMap name

Comment 9 XiuJuan Wang 2021-07-16 10:04:38 UTC
Verified on 4.9.0-0.nightly-2021-07-15-212414 cluster

Comment 12 errata-xmlrpc 2021-10-18 17:38:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.