Description of problem: Fail to pull private builder image with correct pullSecret: $ oc logs builds/ruby-ex-4 Cloning "https://github.com/openshift/ruby-ex.git" ... Commit: bbb670185b6ce67294cc461ae9c18710e6f26089 (Merge pull request #18 from durandom/master) Author: Ben Parees <bparees.github.com> Date: Thu Dec 7 14:53:36 2017 -0500 pulling image error : repository docker.io/wzheng/ruby-22-centos7 not found: does not exist or no pull access error: build error: unable to get docker.io/wzheng/ruby-22-centos7:latest Version-Release number of selected component (if applicable): 3.10.0-0.16.0 How reproducible: always Steps to Reproduce: 1. Prepare a private builder image in docker hub 2. docker login to get the secret file under /home/wzheng/.docker/config.json 3. Create a pullSecret $ oc create secret generic pull --from-file=.dockerconfigjson=/home/wzheng/.docker/config.json --type=kubernetes.io/dockerconfigjson $ oc secrets link builder pull 4. Create a buildconfig to add pullSecret in: Strategy: Source URL: https://github.com/openshift/ruby-hello-world.git From Image: DockerImage docker.io/wzheng/ruby-22-centos7:latest Pull Secret Name: pull Output to: ImageStreamTag origin-ruby-sample:latest Post Commit Hook: ["<image-entrypoint>", "bundle", "exec", "rake", "test"] 5. Start build Actual results: Cannot pull private image. Expected results: Should pull to build with. Additional info: Same secret file works in 3.9 with same steps.
Wenjing can you provide level 5 build logs please, and also the pod yaml from the builder pod.
I was unable to recreate this using origin master + a private repository on docker.io. Relevant Loglevel 5 output: I0411 13:13:54.880649 1 sti.go:217] With force pull true, setting policies to always I0411 13:13:54.880679 1 cfg.go:39] Locating docker auth for image docker.io/bparees/test and type PULL_DOCKERCFG_PATH I0411 13:13:54.880687 1 cfg.go:49] Getting docker auth in paths : [/var/run/secrets/openshift.io/pull] I0411 13:13:54.880707 1 config.go:131] looking for config.json at /var/run/secrets/openshift.io/pull/config.json I0411 13:13:54.881178 1 cfg.go:62] Using bparees user for Docker authentication for image docker.io/bparees/test I0411 13:13:54.881459 1 util.go:266] Pulling image "docker.io/bparees/test" ...
Build log and builder pod yaml file are in attachment. There is a similar bug which cannot be reproduced in origin either: https://bugzilla.redhat.com/show_bug.cgi?id=1561989 , just for your reference.
Created attachment 1420750 [details] Build log
Created attachment 1420751 [details] Builder pod yaml file
Can you confirm that the host node running your build is able to pull the image by hand (via "docker pull") when you've done a docker login w/ the same credentials?
Wenjing any update on this? (Can you pull the image via "docker pull" executed on the cluster node host manually?)
Ben, sorry for the delay. I have tried to pull from node after docker login, it can be pulled successfully.
And same config.json file with same private build on v3.9.22 env works well, image can be pulled to build.
> And same config.json file with same private build on v3.9.22 env works well, > image can be pulled to build. Am I understanding that you can make the cluster work if you generate the config.json from the host node (and then create a secret from it) but not when you generate the config.json on your client machine? If so what is the delta between the config.json that doesn't work and the one that does?
No, the config.json file are the same. Le me make it more clear here: 1. On node, I can docker pull the private image my docker hub account which is the same with item #2 and item #3; 2. In my client side, I can use config.json to pull private image with OCP 3.9 env; 3. In my client side (same client with item #2), I cannot pull private image like the bug describes.
do you possibly have multiple entries for docker.io in your config.json?
No, I have only one account for docker.io, so no multiple entries.
Can you provide your config.json? You can redact the auth values.
I will send my account to you in an email : )
i'm still unable to recreate this on origin (using your credentials, thanks for sending them). Before I go setting up an OCP cluster, is it readily recreatable on other OCP v3.10 clusters?
Can you try the workaround described in comment 27 here: https://bugzilla.redhat.com/show_bug.cgi?id=1561989#c27 namely, edit your /etc/sysconfig/docker to include: ADD_REGISTRY='-add-registry docker.io --add-registry registry.reg-aws.openshift.com --add-registry registry.access.redhat.com' and restart docker before running your test?
It works if add "--add-registry docker.io" to ADD_REGISTRY=. But it is weird that it can be pulled without this option in node and also 3.9 works without it.
I agree that it is strange, but i think that makes it a containers bug or at least a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1561989 so i'm going to mark it as such. *** This bug has been marked as a duplicate of bug 1561989 ***
(In reply to Ben Parees from comment #19) > I agree that it is strange, but i think that makes it a containers bug or at > least a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1561989 so i'm > going to mark it as such. > > *** This bug has been marked as a duplicate of bug 1561989 *** I think they are different, since bug #1561989 can also be reproduced in 3.9, but this bug cannot.
I believe in v3.9 the builder code used its own vendoring of the docker-engine code so it may not have been vulnerable to the bug. In v3.10 we are using the same code as the rest of k8s. as you can see it does not contain the normalization logic: https://github.com/openshift/origin/blob/release-3.9/pkg/build/builder/vendor/github.com/docker/engine-api/client/image_pull.go in v3.10 the builder images do not vendor their own copy of image_pull, so they are getting this one: https://github.com/openshift/origin/blob/master/vendor/github.com/docker/docker/client/image_pull.go#L23 which has been pointed to as the root issue of 1561989. *** This bug has been marked as a duplicate of bug 1561989 ***