Bug 1649063 - Binary builds are not automatically including the pull secret from the service account during the build.
Summary: Binary builds are not automatically including the pull secret from the servi...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.1.0
Assignee: Ben Parees
QA Contact: wewang
Depends On: 1648775
Blocks: 1649062
TreeView+ depends on / blocked
Reported: 2018-11-12 20:05 UTC by Ben Parees
Modified: 2022-03-13 16:02 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Builds that do not explicitly indicate the docker image they consume (by providing an inline dockerfile or defining the docker strategy's From field) and do not explicitly indicate a pull secret to use, will now use the build's service account's docker secret by default. Examples of such builds would be a build that includes a Dockerfile in a git repo. Reason: Previously these builds would use no secret and potentially fail if the base image was not public. Result: Those builds will now succeed without the need to either explicitly specify a pull secret, or explicitly specify the base image in the buildconfig.
Clone Of: 1648775
Last Closed: 2019-06-04 10:40:51 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:40:59 UTC

Description Ben Parees 2018-11-12 20:05:47 UTC
+++ This bug was initially created as a clone of Bug #1648775 +++

Description of problem: 

You should be able to reproduce the issue by creating a namespace called base and put a docker Image in the namespace called web:latest now you create another namespace let’s call it test and allow the service accounts in test namespace to pull images from the base namespace.
In the test namespace you start a binary build that has a dockerfile with FROM docker-registry.default.svc:5000/base/web

The partial build success results from the fact that the requested image is locally available in the docker pool.

A workaround is to add the pull secret of the builder service account to the dockerStrategy like this:
You have to make sure that the image is not already pulled on the node where your build is running.

A workaround is to add the pull secret of the builder service account to the dockerStrategy like this:
        name: builder-dockercfg-1775b

another workaround is to reference the ImageStreamTag (this will override the first FROM in the Dockerfile) like this:
        kind: ImageStreamTag
        name: 'mip-web:latest'
        namespace: base

The workarounds are not feasible as the user need to change all the pipelines to incorporate the workaround.

Anyway the normal k8s behavior is that pullSecret is mounted in the pod (https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#service-account-admission-controller)
3. If the pod does not contain any ImagePullSecrets, then ImagePullSecrets of the ServiceAccount should be added to the pod.
1] Nov 06 11:05:46 master.example.com atomic-openshift-master-api[50426]: I1106 11:05:46.400209   50426 rbac.go:116] RBAC DENY: user "system:anonymous" groups ["system:unauthenticated"] cannot "get" resource "imagestreams/layers.image.openshift.io" named "mip-web" in namespace "base"

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
oc new-project base
oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git -n base

cat <<EOF | oc create -n base -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
  name: image-pull-access-fptest
  namespace: base
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: 'system:image-puller'
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: 'system:serviceaccounts:fptest'

oc new-project fptest
oc create is myimage -n fptest

cat <<EOF | oc create -n fptest -f -
apiVersion: build.openshift.io/v1
kind: BuildConfig
  name: binary-build
  namespace: fptest
  failedBuildsHistoryLimit: 5
      kind: ImageStreamTag
      name: 'myimage:latest'
  postCommit: {}
  resources: {}
  runPolicy: Serial
    binary: {}
    type: Binary
    dockerStrategy: {}
    type: Docker
  successfulBuildsHistoryLimit: 5
  triggers: []

echo "FROM docker-registry.default.svc:5000/base/ruby-25-centos7" > Dockerfile

oc start-build binary-build --from-file=Dockerfile -n fptest

Actual results: Binary builds doesnt automatically include the pull secret from the service account during the build.

Expected results: We expect the build pod has mounted the pullSecret of its SA, even when it is not explicitly configured (https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#service-account-admission-controller point 3):
"If the pod does not contain any ImagePullSecrets, then ImagePullSecrets of the ServiceAccount are added to the pod."

Additional info:

--- Additional comment from Ben Parees on 2018-11-12 09:21:11 EST ---

Does this occur w/ a non-binary build?

--- Additional comment from Ben Parees on 2018-11-12 15:04:01 EST ---

The k8s documented behavior has no bearing on this because builds must do their own determination of what secret to use because builds can reference images in different ways than pods do, and the secrets that k8s mounts into the build pod are not the secrets the build will use.

I do not think this is specific to binary builds.  Any build where you have not explicitly told us in the buildconfig, what image you will be referencing, will have this issue.

The build must make the decision, prior to starting the build pod, what secrets to mount into the build pod so that docker can pull the images.  Since we don't know what your dockerfile looks like until after the build pod starts (at which point the dockerfile is either streamed in (binary build) or git cloned(normal build)), we can't mount it in advance.

Yes, it is technically possible that we can fallback to using the builder service account's docker secret in this scenario, but that will only solve the problem for builds that reference the internal registry.  You would still have this problem if you referenced some other private registry.

As you've noted, the solutions are either:

1) explicitly indicate the image you want to reference via the buildconfig's From field
2) explicitly indicate the secret you want to use to pull the image via the buildconfig's pullsecret field.

I will work an update that causes us to fallback to the service account's docker secret(if available) when we have no other information about what image is being pulled (as is the case here), but you will still have problems if you reference some other private registry in this way, so it would really be better to solve this in what I consider to be the correct way:  reference a secret you know can pull the image.

--- Additional comment from Ben Parees on 2018-11-12 15:04:40 EST ---


Comment 1 Ben Parees 2018-11-12 20:18:36 UTC

Comment 2 Ben Parees 2018-11-30 12:58:20 UTC
this has been in modified for 2 weeks.  whatever script is supposed to be moving things to on_qa is not working.

Comment 3 wewang 2018-12-03 03:19:05 UTC
Verified in openshift: v4.0.0-0.80.0

Comment 6 errata-xmlrpc 2019-06-04 10:40:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.