Bug 1565944 - Fail to pull private builder image with correct pullSecret
Summary: Fail to pull private builder image with correct pullSecret
Keywords:
Status: CLOSED DUPLICATE of bug 1561989
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.10.0
Assignee: Ben Parees
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-11 06:41 UTC by Wenjing Zheng
Modified: 2018-04-25 14:28 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-25 14:28:19 UTC
Target Upstream Version:


Attachments (Terms of Use)
Build log (12.56 KB, text/plain)
2018-04-12 07:56 UTC, Wenjing Zheng
no flags Details
Builder pod yaml file (13.51 KB, text/plain)
2018-04-12 07:57 UTC, Wenjing Zheng
no flags Details

Description Wenjing Zheng 2018-04-11 06:41:27 UTC
Description of problem:
Fail to pull private builder image with correct pullSecret:
$ oc logs builds/ruby-ex-4
Cloning "https://github.com/openshift/ruby-ex.git" ...
    Commit:    bbb670185b6ce67294cc461ae9c18710e6f26089 (Merge pull request #18 from durandom/master)
    Author:    Ben Parees <bparees@users.noreply.github.com>
    Date:    Thu Dec 7 14:53:36 2017 -0500
pulling image error : repository docker.io/wzheng/ruby-22-centos7 not found: does not exist or no pull access
error: build error: unable to get docker.io/wzheng/ruby-22-centos7:latest


Version-Release number of selected component (if applicable):
3.10.0-0.16.0

How reproducible:
always

Steps to Reproduce:
1. Prepare a private builder image in docker hub
2. docker login to get the secret file under /home/wzheng/.docker/config.json
3. Create a pullSecret
$ oc  create secret generic pull  --from-file=.dockerconfigjson=/home/wzheng/.docker/config.json  --type=kubernetes.io/dockerconfigjson
$ oc secrets link builder pull
4. Create a buildconfig to add pullSecret in:
Strategy:        Source
URL:            https://github.com/openshift/ruby-hello-world.git
From Image:        DockerImage docker.io/wzheng/ruby-22-centos7:latest
Pull Secret Name:    pull
Output to:        ImageStreamTag origin-ruby-sample:latest
Post Commit Hook:    ["<image-entrypoint>", "bundle", "exec", "rake", "test"]
5. Start build

Actual results:
Cannot pull private image.

Expected results:
Should pull to build with.

Additional info:
Same secret file works in 3.9 with same steps.

Comment 1 Ben Parees 2018-04-11 12:07:22 UTC
Wenjing can you provide level 5 build logs please, and also the pod yaml from the builder pod.

Comment 2 Ben Parees 2018-04-11 13:16:44 UTC
I was unable to recreate this using origin master + a private repository on docker.io.  Relevant Loglevel 5 output:

I0411 13:13:54.880649       1 sti.go:217] With force pull true, setting policies to always
I0411 13:13:54.880679       1 cfg.go:39] Locating docker auth for image docker.io/bparees/test and type PULL_DOCKERCFG_PATH
I0411 13:13:54.880687       1 cfg.go:49] Getting docker auth in paths : [/var/run/secrets/openshift.io/pull]
I0411 13:13:54.880707       1 config.go:131] looking for config.json at /var/run/secrets/openshift.io/pull/config.json
I0411 13:13:54.881178       1 cfg.go:62] Using bparees user for Docker authentication for image docker.io/bparees/test
I0411 13:13:54.881459       1 util.go:266] Pulling image "docker.io/bparees/test" ...

Comment 3 Wenjing Zheng 2018-04-12 07:55:53 UTC
Build log and builder pod yaml file are in attachment.

There is a similar bug which cannot be reproduced in origin either: https://bugzilla.redhat.com/show_bug.cgi?id=1561989 , just for your reference.

Comment 4 Wenjing Zheng 2018-04-12 07:56:27 UTC
Created attachment 1420750 [details]
Build log

Comment 5 Wenjing Zheng 2018-04-12 07:57:07 UTC
Created attachment 1420751 [details]
Builder pod yaml file

Comment 6 Ben Parees 2018-04-12 15:24:18 UTC
Can you confirm that the host node running your build is able to pull the image by hand (via "docker pull") when you've done a docker login w/ the same credentials?

Comment 7 Ben Parees 2018-04-18 12:29:06 UTC
Wenjing any update on this?  (Can you pull the image via "docker pull" executed on the cluster node host manually?)

Comment 8 Wenjing Zheng 2018-04-19 10:00:24 UTC
Ben, sorry for the delay. I have tried to pull from node after docker login, it can be pulled successfully.

Comment 9 Wenjing Zheng 2018-04-19 10:05:55 UTC
And same config.json file with same private build on v3.9.22 env works well, image can be pulled to build.

Comment 10 Ben Parees 2018-04-19 12:57:33 UTC
> And same config.json file with same private build on v3.9.22 env works well, 
> image can be pulled to build.

Am I understanding that you can make the cluster work if you generate the config.json from the host node (and then create a secret from it) but not when you generate the config.json on your client machine?

If so what is the delta between the config.json that doesn't work and the one that does?

Comment 11 Wenjing Zheng 2018-04-20 02:47:12 UTC
No, the config.json file are the same. Le me make it more clear here:
1. On node, I can docker pull the private image my docker hub account which is the same with item #2 and item #3;
2. In my client side, I can use config.json to pull private image with OCP 3.9 env;
3. In my client side (same client with item #2), I cannot pull private image like the bug describes.

Comment 12 Ben Parees 2018-04-20 03:17:14 UTC
do you possibly have multiple entries for docker.io in your config.json?

Comment 13 Wenjing Zheng 2018-04-20 04:49:07 UTC
No, I have only one account for docker.io, so no multiple entries.

Comment 14 Ben Parees 2018-04-20 13:18:21 UTC
Can you provide your config.json?  You can redact the auth values.

Comment 15 Wenjing Zheng 2018-04-23 07:24:23 UTC
I will send my account to you in an email : )

Comment 16 Ben Parees 2018-04-23 22:31:21 UTC
i'm still unable to recreate this on origin (using your credentials, thanks for sending them).  Before I go setting up an OCP cluster, is it readily recreatable on other OCP v3.10 clusters?

Comment 17 Ben Parees 2018-04-24 03:50:30 UTC
Can you try the workaround described in comment 27 here:
https://bugzilla.redhat.com/show_bug.cgi?id=1561989#c27

namely, edit your /etc/sysconfig/docker to include:

ADD_REGISTRY='-add-registry docker.io --add-registry registry.reg-aws.openshift.com --add-registry registry.access.redhat.com'


and restart docker before running your test?

Comment 18 Wenjing Zheng 2018-04-24 10:26:18 UTC
It works if add "--add-registry docker.io" to ADD_REGISTRY=. But it is weird that it can be pulled without this option in node and also 3.9 works without it.

Comment 19 Ben Parees 2018-04-24 12:56:19 UTC
I agree that it is strange, but i think that makes it a containers bug or at least a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1561989 so i'm going to mark it as such.

*** This bug has been marked as a duplicate of bug 1561989 ***

Comment 20 Wenjing Zheng 2018-04-25 06:16:10 UTC
(In reply to Ben Parees from comment #19)
> I agree that it is strange, but i think that makes it a containers bug or at
> least a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1561989 so i'm
> going to mark it as such.
> 
> *** This bug has been marked as a duplicate of bug 1561989 ***

I think they are different, since bug #1561989 can also be reproduced in 3.9, but this bug cannot.

Comment 21 Ben Parees 2018-04-25 14:28:19 UTC
I believe in v3.9 the builder code used its own vendoring of the docker-engine code so it may not have been vulnerable to the bug.  In v3.10 we are using the same code as the rest of k8s.

as you can see it does not contain the normalization logic:
https://github.com/openshift/origin/blob/release-3.9/pkg/build/builder/vendor/github.com/docker/engine-api/client/image_pull.go


in v3.10 the builder images do not vendor their own copy of image_pull, so they are getting this one:
https://github.com/openshift/origin/blob/master/vendor/github.com/docker/docker/client/image_pull.go#L23

which has been pointed to as the root issue of 1561989.

*** This bug has been marked as a duplicate of bug 1561989 ***


Note You need to log in before you can comment on or make changes to this bug.