Description of problem: will meet "API error (403)" error when pushing image with Docker credentials. Version-Release number of selected component (if applicable): oc v3.0.1.0-388-g51e9da7 kubernetes v1.0.0 docker version: Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): ba1f6c3/1.6.2 OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): ba1f6c3/1.6.2 OS/Arch (server): linux/amd64 How reproducible: always Steps to Reproduce: 1. Setup the OpenShift V3 environment according to: OpenShift 3 Application Lifecycle Sample and login to server $ oc login --server=$master -u $user $ oc new-project $project-name 2. Get the '.dockercfg' file which contain account credential 1) register an account on DockerHub 2) log in to the registry to get the account credential $ docker login ## the credential file named ".dockercfg" will be generated under $HOME directory after login successfully 3. Create the secret $ oc secrets new pushme path/to/.dockercfg 4. Add the secret "pushme" to builder service account $ oc secrets add serviceaccount/builder secrets/pushme 5. Update buildconfig to relate above secret $ oc edit buildconfig/$bc-name <----------snip---------> { "kind": "ImageStream", "apiVersion": "v1beta1", "metadata": { "name": "origin-ruby-sample" }, "spec":{ "dockerImageRepository": "chunyunchen/origin-ruby-sample" }, "parameters": { "output": { "to": { "kind": "ImageStreamTag", "name": "origin-ruby-sample:latest" }, "pushSecret":{ "name":"pushme" ## the "pushSecretName" is the same with above secret name } }, <----------snip---------> "strategy": { "type": "Source" } }, <----------snip---------> 6. Create the application to trigger build and wait the build to finish $ oc process -f template.json | oc create -f - 7. Check the build logs $ oc get build $ oc build-logs ruby-sample-build-1 Actual results: <---------------snip---------------> I0722 02:13:53.788552 1 docker.go:521] Committing container with config: {Hostname: Domainname: User: Memory:0 MemorySwap:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:false AttachStderr:false PortSpecs:[] ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[OPENSHIFT_BUILD_NAME=ruby-sample-build-2 OPENSHIFT_BUILD_NAMESPACE=chunp OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/sti-python BUILD_LOGLEVEL=5] Cmd:[/usr/local/sti/run] DNS:[] Image: Volumes:map[] VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Labels:map[io.openshift.s2i.build.image:registry.access.redhat.com/openshift3/python-33-rhel7:latest io.openshift.s2i.build.commit.author:Michal Fojtik <mi> io.openshift.s2i.build.commit.id:6755bbc io.openshift.s2i.build.commit.ref:master io.k8s.display-name:chunyunchen/python-sample-722:latest io.openshift.s2i.build.commit.date:Wed Jul 15 17:55:59 2015 +0200 io.openshift.s2i.build.commit.message:Merge pull request #46 from mfojtik/fix-build-script io.openshift.s2i.build.source-location:https://github.com/openshift/sti-python io.openshift.s2i.build.source-context-dir:3.3/test/standalone-test-app]} I0722 02:13:56.634909 1 sti.go:269] Successfully built chunyunchen/python-sample-722:latest I0722 02:13:57.968547 1 cleanup.go:23] Removing temporary directory /tmp/sti027940679 I0722 02:13:57.968601 1 fs.go:99] Removing directory '/tmp/sti027940679' I0722 02:13:57.969189 1 cfg.go:78] Found Docker authentication configuration in '/root/.dockercfg' I0722 02:13:57.969203 1 cfg.go:46] PUSH_DOCKERCFG_PATH=/var/run/secrets/openshift.io/push/.dockercfg I0722 02:13:57.969208 1 cfg.go:78] Found Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg' I0722 02:13:57.969232 1 cfg.go:50] Problem accessing /var/run/secrets/openshift.io/push/.dockercfg: stat /var/run/secrets/openshift.io/push/.dockercfg: no such file or directory I0722 02:13:57.969251 1 sti.go:148] Pushing chunyunchen/python-sample-722:latest image ... E0722 02:13:57.972207 1 dockerutil.go:50] push for image chunyunchen/python-sample-722:latest failed, will retry in 10s ... E0722 02:14:07.981425 1 dockerutil.go:50] push for image chunyunchen/python-sample-722:latest failed, will retry in 10s ... F0722 02:14:18.678584 1 builder.go:64] Build error: Failed to push image: API error (403): Error: Status 403 trying to push repository chunyunchen/python-sample-722 to official registry: needs to be forced Expected results: Should push image with Docker credentials successfully. Additional info: the same bug on OpenShift Origin: https://bugzilla.redhat.com/show_bug.cgi?id=1224210
I0722 02:13:57.969232 1 cfg.go:50] Problem accessing /var/run/secrets/openshift.io/push/.dockercfg: stat /var/run/secrets/openshift.io/push/.dockercfg: no such file or directory Looks like the issue with secrets and older versions of Docker. Please ensure that you have a recent 1.6.2 version
Does pushing to the local registry work?
It can be pushed to the local registry, the logs like below: I0722 03:24:56.089769 1 sti.go:269] Successfully built 172.30.234.245:5000/chunp/origin-ruby-sample:latest I0722 03:24:57.433893 1 cleanup.go:23] Removing temporary directory /tmp/sti374841799 I0722 03:24:57.433920 1 fs.go:99] Removing directory '/tmp/sti374841799' I0722 03:24:57.434480 1 cfg.go:78] Found Docker authentication configuration in '/root/.dockercfg' I0722 03:24:57.434495 1 cfg.go:46] PUSH_DOCKERCFG_PATH=/var/run/secrets/openshift.io/push/.dockercfg I0722 03:24:57.434501 1 cfg.go:78] Found Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg' I0722 03:24:57.434799 1 cfg.go:64] Using serviceaccount user for Docker authentication I0722 03:24:57.434841 1 sti.go:145] Using provided push secret for pushing 172.30.234.245:5000/chunp/origin-ruby-sample:latest image I0722 03:24:57.434850 1 sti.go:148] Pushing 172.30.234.245:5000/chunp/origin-ruby-sample:latest image ... Image successfully pushed Digest: sha256:f8699168b06d66cc6b9bb03cfccdbdfdba2ef93cbc8e7f49dd554c85034fdd0a I0722 03:26:56.495641 1 sti.go:152] Successfully pushed 172.30.234.245:5000/chunp/origin-ruby-sample:latest
This is working for me on Fedora but not on RHEL. RHEL is ignoring the --confirm-def-push=false flag. And there's a bug open for that: https://bugzilla.redhat.com/show_bug.cgi?id=1241952
The workaround for this is to edit /var/lib/docker/repositories-devicemapper: Change the value of "ConfirmDefPush" from true to false. Restart the Docker daemon. It is fixed in docker 1.7 for RHEL which will be released shortly. Given the workaround and that it will be fixed shortly, I'm lowering the severity.
The issue is still reproduced when docker is 1.7.1 and the value of "ConfirmDefPush" is true [root@openshift-138 ~]# docker version Client version: 1.7.1 Client API version: 1.19 Package Version (client): docker-1.7.1-108.el7.x86_64 Go version (client): go1.4.2 Git commit (client): 3043001/1.7.1 OS/Arch (client): linux/amd64 Server version: 1.7.1 Server API version: 1.19 Package Version (server): docker-1.7.1-108.el7.x86_64 Go version (server): go1.4.2 Git commit (server): 3043001/1.7.1 OS/Arch (server): linux/amd64
--confirm-def-push is a custom addition to the RHEL Docker RPM. There is no ability currently for us to confirm that it's ok to push to the Hub in the Docker client API that we use when pushing after a completed build. I'm not sure there's anything we can do here.
Andy, do we document somewhere that the --confirm-def-push needs to be turned off in order for push to docker hub to work?
Cesar - I don't know offhand.
chunchen looking at the history of this bug, not sure if there is really a bug anymore. The version of docker we have in RHEL requires you to have the --confirm-def-push=false flag set on the daemon so that we can push to the DockerHub. At some point that flag was broken and you had to manually edit the repositories-devicemapper file to get the same effect. That part of it should be fixed, but you still need to specify the flag.
chunchen can you please try again with the --confirm-def-push=false flag set on your host's docker daemon? thanks.
It works well with the --confirm-def-push=false flag set on docker daemon. I1027 06:55:55.713080 1 sti.go:296] Successfully built docker.io/chunyunchen/origin-ruby-sample-sti2:latest I1027 06:56:04.547556 1 cleanup.go:23] Removing temporary directory /tmp/s2i-build531069471 I1027 06:56:04.555606 1 fs.go:99] Removing directory '/tmp/s2i-build531069471' I1027 06:56:04.566829 1 sti.go:210] Using provided push secret for pushing docker.io/chunyunchen/origin-ruby-sample-sti2:latest image I1027 06:56:04.566852 1 sti.go:214] Pushing docker.io/chunyunchen/origin-ruby-sample-sti2:latest image ... I1027 07:00:27.301472 1 sti.go:230] Successfully pushed docker.io/chunyunchen/origin-ruby-sample-sti2:latest
This fix is available in OpenShift Enterprise 3.1.