Bug 1565944
| Summary: | Fail to pull private builder image with correct pullSecret | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wenjing Zheng <wzheng> | ||||||
| Component: | Build | Assignee: | Ben Parees <bparees> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Wenjing Zheng <wzheng> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 3.10.0 | CC: | aos-bugs, wzheng | ||||||
| Target Milestone: | --- | Keywords: | Regression, Reopened | ||||||
| Target Release: | 3.10.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-04-25 14:28:19 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Wenjing Zheng
2018-04-11 06:41:27 UTC
Wenjing can you provide level 5 build logs please, and also the pod yaml from the builder pod. I was unable to recreate this using origin master + a private repository on docker.io. Relevant Loglevel 5 output: I0411 13:13:54.880649 1 sti.go:217] With force pull true, setting policies to always I0411 13:13:54.880679 1 cfg.go:39] Locating docker auth for image docker.io/bparees/test and type PULL_DOCKERCFG_PATH I0411 13:13:54.880687 1 cfg.go:49] Getting docker auth in paths : [/var/run/secrets/openshift.io/pull] I0411 13:13:54.880707 1 config.go:131] looking for config.json at /var/run/secrets/openshift.io/pull/config.json I0411 13:13:54.881178 1 cfg.go:62] Using bparees user for Docker authentication for image docker.io/bparees/test I0411 13:13:54.881459 1 util.go:266] Pulling image "docker.io/bparees/test" ... Build log and builder pod yaml file are in attachment. There is a similar bug which cannot be reproduced in origin either: https://bugzilla.redhat.com/show_bug.cgi?id=1561989 , just for your reference. Created attachment 1420750 [details]
Build log
Created attachment 1420751 [details]
Builder pod yaml file
Can you confirm that the host node running your build is able to pull the image by hand (via "docker pull") when you've done a docker login w/ the same credentials? Wenjing any update on this? (Can you pull the image via "docker pull" executed on the cluster node host manually?) Ben, sorry for the delay. I have tried to pull from node after docker login, it can be pulled successfully. And same config.json file with same private build on v3.9.22 env works well, image can be pulled to build. > And same config.json file with same private build on v3.9.22 env works well,
> image can be pulled to build.
Am I understanding that you can make the cluster work if you generate the config.json from the host node (and then create a secret from it) but not when you generate the config.json on your client machine?
If so what is the delta between the config.json that doesn't work and the one that does?
No, the config.json file are the same. Le me make it more clear here: 1. On node, I can docker pull the private image my docker hub account which is the same with item #2 and item #3; 2. In my client side, I can use config.json to pull private image with OCP 3.9 env; 3. In my client side (same client with item #2), I cannot pull private image like the bug describes. do you possibly have multiple entries for docker.io in your config.json? No, I have only one account for docker.io, so no multiple entries. Can you provide your config.json? You can redact the auth values. I will send my account to you in an email : ) i'm still unable to recreate this on origin (using your credentials, thanks for sending them). Before I go setting up an OCP cluster, is it readily recreatable on other OCP v3.10 clusters? Can you try the workaround described in comment 27 here: https://bugzilla.redhat.com/show_bug.cgi?id=1561989#c27 namely, edit your /etc/sysconfig/docker to include: ADD_REGISTRY='-add-registry docker.io --add-registry registry.reg-aws.openshift.com --add-registry registry.access.redhat.com' and restart docker before running your test? It works if add "--add-registry docker.io" to ADD_REGISTRY=. But it is weird that it can be pulled without this option in node and also 3.9 works without it. I agree that it is strange, but i think that makes it a containers bug or at least a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1561989 so i'm going to mark it as such. *** This bug has been marked as a duplicate of bug 1561989 *** (In reply to Ben Parees from comment #19) > I agree that it is strange, but i think that makes it a containers bug or at > least a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1561989 so i'm > going to mark it as such. > > *** This bug has been marked as a duplicate of bug 1561989 *** I think they are different, since bug #1561989 can also be reproduced in 3.9, but this bug cannot. I believe in v3.9 the builder code used its own vendoring of the docker-engine code so it may not have been vulnerable to the bug. In v3.10 we are using the same code as the rest of k8s. as you can see it does not contain the normalization logic: https://github.com/openshift/origin/blob/release-3.9/pkg/build/builder/vendor/github.com/docker/engine-api/client/image_pull.go in v3.10 the builder images do not vendor their own copy of image_pull, so they are getting this one: https://github.com/openshift/origin/blob/master/vendor/github.com/docker/docker/client/image_pull.go#L23 which has been pointed to as the root issue of 1561989. *** This bug has been marked as a duplicate of bug 1561989 *** |