Hide Forgot
Created attachment 1229052 [details] oc describe pods Description of problem: User reported pulling images from private Docker Hub repo is failing; quoting user report: -- oc new-app eupraxialabs/389ds:1.3.4.0 --name=ldap-389ds [dbrewer@controller1 389-ds]$ oc secrets link builder private-repos --for=pull [dbrewer@controller1 389-ds]$ oc secrets link default private-repos --for=pull [dbrewer@controller1 389-ds]$ oc secrets link deployer private-repos --for=pull [dbrewer@controller1 389-ds]$ oc describe secret private-repos Name: private-repos Namespace: cloud-management Labels: <none> Annotations: <none> Type: kubernetes.io/dockerconfigjson Data ==== .dockerconfigjson: 2296 bytes That Docker config JSON file is used on the desktop to pull images successfully to a Docker Host. It's failing on Next Gen. Failed Failed to pull image "eupraxialabs/389ds@sha256:15595915f3d50ac4f4ddd090158eb85930c4f5250a2fca1b517353cb114b4bbb": [unauthorized: authentication required, unauthorized: authentication required] 5 times in the last 4 minutes -- I was able to induce the same: Version-Release number of selected component (if applicable): oc version oc v3.3.0.35 kubernetes v1.3.0+52492b4 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://api.preview.openshift.com:443 openshift v3.3.1.3 kubernetes v1.3.0+52492b4 How reproducible: always Steps to Reproduce: 1. The needed pull secret has been created using 'oc secrets new pull-secret .dockerconfigjson=/home/jfiala/.docker/config.json': -- oc describe secret pull-secret Name: pull-secret Namespace: yap Labels: <none> Annotations: <none> Type: kubernetes.io/dockerconfigjson Data ==== .dockerconfigjson: 210 bytes -- The behavior is the same if credentials are passed to the dokcercfg secret manually, rather than importing from an existing .docker/config.json. 2. The secret is linked to the default service account: -- oc secret link default pull-secret --for=pull -- 3. New app is created using an existing private repo on dockerhub: -- oc new-app fiala82/testpriv --> Found Docker image 462d623 (22 hours old) from Docker Hub for "fiala82/testpriv" * An image stream will be created as "testpriv:latest" that will track this image * This image will be deployed in deployment config "testpriv" * The image does not expose any ports - if you want to load balance or send traffic to this component you will need to create a service with 'expose dc/testpriv --port=[port]' later * WARNING: Image "fiala82/testpriv" runs as the 'root' user which may not be permitted by your cluster administrator --> Creating resources with label app=testpriv ... imagestream "testpriv" created deploymentconfig "testpriv" created --> Success Run 'oc status' to view your app. -- Actual results: I get the same authentication warning as reported by the user: -- Failed to pull image "fiala82/testpriv@sha256:9bdee97d66cdbc8344e8ae1116fabb63a85e31758efd98f0b7ae61badb75ebf7": unauthorized: authentication required -- Full event list is attached in events.txt. Ultimately, the deployment fails with 'error: update acceptor rejected testpriv-1: pods for deployment "testpriv-1" took longer than 600 seconds to become ready'. Expected results: No auth issues when pulling images from a private dockerhub repo, providing the dockerconfig secret is set up properly and linked to the 'default' service account.
Created attachment 1229053 [details] events
Can you verify the secret was actually linked to the SA? oc get sa default
FYI, I tried to reproduce this locally on 1.4 (master) and it worked fine. I mean I created the pull secret for my DockerHub and then kubelet was able to pull the image (it gives me auth error before). I wonder if this might be a timing issue or something that is broken on 3.3 (have to try there).
This was induced on Online Dev Preview (OpenShift Master: v3.3.1.3) I can confirm the secret is linked: -- oc get sa default NAME SECRETS AGE default 3 22d oc describe sa default Name: default Namespace: yap Labels: <none> Image pull secrets: default-dockercfg-ej7ui gitlab-manual pull-secret Mountable secrets: default-token-07vuo default-dockercfg-ej7ui gitlab-manual2 another-pull-secret Tokens: default-token-07vuo default-token-xtrbe --
From the user, it would appear that the secrets issue is only apparent when using from CLI tools: --- I went ahead and created a template with a DockerHub pull secret and it works fine for GitHub and GitLab. objects: - apiVersion: v1 kind: ImageStream metadata: name: openerp spec: dockerImageRepository: registry.gitlab.com/eupraxialabs/openerp pullsecret: name: DockerHub or docker.io/eupraxialabs/openerp ---
Jiri: Can you check if the ImageStream you creates has the right pull secret attached?
I suspect a problem with dockercfg registry name. Could you please provide the keys (registry names) in the dockercfg in question? You can get it for example with: oc describe secret <SECRET_NAME> | sed -n 's/^\.dockercfg:\s*//p' | jq 'keys' There was a bug in docker which caused dockercfg to be created with improper keys (something that docker upstream wouldn't create) and OpenShift was unable to match them against docker.io. My keys in the dockercfg look like this: $ oc describe secret miminar-docker-hub-cfg | sed -n 's/^\.dockercfg:\s*//p' | jq 'keys' [ "https://index.docker.io/v1/" ] I've requested accounts "OpenShift Online (Next Gen) Developer Preview" and "OpenShift Online 3 Developer Preview" accounts to verify there.
@jfiala: I've followed the steps (on recent master, but it doesn't differ that much from what was release in 3.4) and it just works with a few different possibilities how to pass image name. As soon as I unlink the secret and re-try the new-app it won't run. Linking it back and re-trying the deployment worked as well.
I've additionally verified this on: $ oc version oc v1.3.1+ac1d579 kubernetes v1.3.0+52492b4 features: Basic-Auth Server https://localhost:8443 openshift v1.3.1+ac1d579 kubernetes v1.3.0+52492b4 and it's working as expected, following the steps described in the first comment.
@miminar: The secret was created from .docker/config.json (that config works for docker daemon when pulling the image in question); 'describe' produces this: -- oc describe secret pull-secret Name: pull-secret Namespace: yap Labels: <none> Annotations: <none> Type: kubernetes.io/dockerconfigjson Data ==== .dockerconfigjson: 210 bytes -- I have re-created secret by passing the credentials using: oc secret new-dockercfg pull-secret- --docker-username=.. etc. Then, the registry server is as expected: -- oc describe secret pull-secret | sed -n 's/^\.dockercfg:\s*//p' | jq 'keys' [ "https://index.docker.io/v1/" ] -- I have linked the new secret, as well as the kubernetes.io/dockerconfigjson one, to the default sa and re-tested, with the same result. @mfojtik: Not sure how can I tell that. Here's the image stream: -- apiVersion: v1 kind: ImageStream metadata: name: testpriv namespace: yap selfLink: /oapi/v1/namespaces/yap/imagestreams/testpriv uid: ad3d8ba5-bd27-11e6-9b27-0e63b9c1c48f resourceVersion: '497311049' generation: 2 creationTimestamp: '2016-12-08T09:21:18Z' labels: app: testpriv annotations: openshift.io/generated-by: OpenShiftNewApp openshift.io/image.dockerRepositoryCheck: '2016-12-08T09:21:19Z' spec: tags: - name: latest annotations: openshift.io/imported-from: fiala82/testpriv from: kind: DockerImage name: fiala82/testpriv generation: 2 importPolicy: status: dockerImageRepository: '172.30.47.227:5000/yap/testpriv' tags: - tag: latest items: - created: '2016-12-08T09:21:19Z' dockerImageReference: 'fiala82/testpriv@sha256:e32c748ff75dde4f5e6bdf9a3ff53431f43f5629cc3e7f2b09329ac5f30205c3' image: 'sha256:e32c748ff75dde4f5e6bdf9a3ff53431f43f5629cc3e7f2b09329ac5f30205c3' generation: 2 --
I'm still unable to reproduce locally. But I've verified the error with both Server https://api.dev-preview-stg.openshift.com:443 openshift v3.3.1.4 kubernetes v1.3.0+52492b4 and Server https://api.preview.openshift.com:443 openshift v3.3.1.3 kubernetes v1.3.0+52492b4 With the latter, I wasn't able to get past the image import when using `oc secrets new pull-secret`. The error was: $ oc describe pods ... 29s 11s 2 {kubelet ip-172-31-8-200.ec2.internal} spec.containers{registry} Normal Pulling pulling image "miminar/registry:1.2.3.4" 27s 10s 2 {kubelet ip-172-31-8-200.ec2.internal} spec.containers{registry} Warning Failed Failed to pull image "miminar/registry:1.2.3.4": Error: image miminar/registry not found 27s 10s 2 {kubelet ip-172-31-8-200.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "registry" with ErrImagePull: "Error: image miminar/registry not found" When I used `oc secrets new-dockercfg`. I got the same error as listed above and the same as on https://api.dev-preview-stg.openshift.com:443. Can we get registry log from `https://api.preview.openshift.com:443` cluster mentioning at least one of the following images: - miminar/registry - fiala82/testpriv - eupraxialabs/389ds ? Also what's the version docker daemon there?
Marking this as upcoming release after talking with Maciej. We can't reproduce this locally and probably will need some more data from online to see what is going on. Also there seems to be at least one workaround for the moment. Will, can you please describe how exactly the issue was fixed for the user? He added the secrets into a template (but he also have to link them into the default SA I assume).
I've additionally verified the steps against OSE v3.4.0.33+59c4d51-1 works as expected. With all that it looks like the problem is with the online environment. I'm moving this to Online team, since it's something online team should address.
OpenShift Online Preview has been decommissioned, go to https://manage.openshift.com/ for using OpenShift Online starter cluster