Bug 1245472 - Met "API error (403)" when pushing image with Docker credentials
Summary: Met "API error (403)" when pushing image with Docker credentials
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Cesar Wong
QA Contact: chunchen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-22 07:18 UTC by chunchen
Modified: 2016-09-30 02:16 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1224210
Environment:
Last Closed: 2015-11-23 14:24:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description chunchen 2015-07-22 07:18:34 UTC
Description of problem:
will meet "API error (403)" error when pushing image with Docker credentials.

Version-Release number of selected component (if applicable):
oc v3.0.1.0-388-g51e9da7
kubernetes v1.0.0

docker version:
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): ba1f6c3/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): ba1f6c3/1.6.2
OS/Arch (server): linux/amd64

How reproducible:
always

Steps to Reproduce:
1. Setup the OpenShift V3 environment according to: OpenShift 3 Application Lifecycle Sample  and login to server
$ oc login --server=$master -u $user
$ oc new-project $project-name

2. Get the '.dockercfg' file which contain account credential
1) register an account on DockerHub
2) log in to the registry to get the account credential
$ docker login ## the credential file named ".dockercfg" will be generated under $HOME directory after login successfully

3. Create the secret
$ oc secrets new pushme path/to/.dockercfg

4. Add the secret "pushme" to builder service account
$ oc secrets add serviceaccount/builder secrets/pushme

5. Update buildconfig to relate above secret
$ oc edit buildconfig/$bc-name
     <----------snip--------->
       {
         "kind": "ImageStream",
         "apiVersion": "v1beta1",
         "metadata": {
           "name": "origin-ruby-sample"
         },
         "spec":{
         "dockerImageRepository": "chunyunchen/origin-ruby-sample"
       },
             "parameters": {
        "output": {
          "to": {
            "kind": "ImageStreamTag",
            "name": "origin-ruby-sample:latest"
          },
          "pushSecret":{
             "name":"pushme"   ## the "pushSecretName" is the same with above secret name
          }
        },
     <----------snip--------->
        "strategy": {
          "type": "Source"
        }
      },
     <----------snip--------->

6. Create the application to trigger build and wait the build to finish
$ oc process -f template.json | oc create -f -

7. Check the build logs
$ oc get build
$ oc build-logs ruby-sample-build-1

Actual results:
<---------------snip--------------->
I0722 02:13:53.788552       1 docker.go:521] Committing container with config: {Hostname: Domainname: User: Memory:0 MemorySwap:0 CPUShares:0 CPUSet: AttachStdin:false AttachStdout:false AttachStderr:false PortSpecs:[] ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[OPENSHIFT_BUILD_NAME=ruby-sample-build-2 OPENSHIFT_BUILD_NAMESPACE=chunp OPENSHIFT_BUILD_SOURCE=https://github.com/openshift/sti-python BUILD_LOGLEVEL=5] Cmd:[/usr/local/sti/run] DNS:[] Image: Volumes:map[] VolumesFrom: WorkingDir: MacAddress: Entrypoint:[] NetworkDisabled:false SecurityOpts:[] OnBuild:[] Labels:map[io.openshift.s2i.build.image:registry.access.redhat.com/openshift3/python-33-rhel7:latest io.openshift.s2i.build.commit.author:Michal Fojtik <mi> io.openshift.s2i.build.commit.id:6755bbc io.openshift.s2i.build.commit.ref:master io.k8s.display-name:chunyunchen/python-sample-722:latest io.openshift.s2i.build.commit.date:Wed Jul 15 17:55:59 2015 +0200 io.openshift.s2i.build.commit.message:Merge pull request #46 from mfojtik/fix-build-script io.openshift.s2i.build.source-location:https://github.com/openshift/sti-python io.openshift.s2i.build.source-context-dir:3.3/test/standalone-test-app]}
I0722 02:13:56.634909       1 sti.go:269] Successfully built chunyunchen/python-sample-722:latest
I0722 02:13:57.968547       1 cleanup.go:23] Removing temporary directory /tmp/sti027940679
I0722 02:13:57.968601       1 fs.go:99] Removing directory '/tmp/sti027940679'
I0722 02:13:57.969189       1 cfg.go:78] Found Docker authentication configuration in '/root/.dockercfg'
I0722 02:13:57.969203       1 cfg.go:46] PUSH_DOCKERCFG_PATH=/var/run/secrets/openshift.io/push/.dockercfg
I0722 02:13:57.969208       1 cfg.go:78] Found Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I0722 02:13:57.969232       1 cfg.go:50] Problem accessing /var/run/secrets/openshift.io/push/.dockercfg: stat /var/run/secrets/openshift.io/push/.dockercfg: no such file or directory
I0722 02:13:57.969251       1 sti.go:148] Pushing chunyunchen/python-sample-722:latest image ...
E0722 02:13:57.972207       1 dockerutil.go:50] push for image chunyunchen/python-sample-722:latest failed, will retry in 10s ...
E0722 02:14:07.981425       1 dockerutil.go:50] push for image chunyunchen/python-sample-722:latest failed, will retry in 10s ...
F0722 02:14:18.678584       1 builder.go:64] Build error: Failed to push image: API error (403): Error: Status 403 trying to push repository chunyunchen/python-sample-722 to official registry: needs to be forced

Expected results:
Should push image with Docker credentials successfully.

Additional info:
the same bug on OpenShift Origin:
https://bugzilla.redhat.com/show_bug.cgi?id=1224210

Comment 2 Cesar Wong 2015-07-22 18:41:38 UTC
I0722 02:13:57.969232       1 cfg.go:50] Problem accessing /var/run/secrets/openshift.io/push/.dockercfg: stat /var/run/secrets/openshift.io/push/.dockercfg: no such file or directory

Looks like the issue with secrets and older versions of Docker. Please ensure that you have a recent 1.6.2 version

Comment 3 Cesar Wong 2015-07-22 19:44:52 UTC
Does pushing to the local registry work?

Comment 4 chunchen 2015-07-23 02:06:06 UTC
It can be pushed to the local registry, the logs like below:

I0722 03:24:56.089769       1 sti.go:269] Successfully built 172.30.234.245:5000/chunp/origin-ruby-sample:latest
I0722 03:24:57.433893       1 cleanup.go:23] Removing temporary directory /tmp/sti374841799
I0722 03:24:57.433920       1 fs.go:99] Removing directory '/tmp/sti374841799'
I0722 03:24:57.434480       1 cfg.go:78] Found Docker authentication configuration in '/root/.dockercfg'
I0722 03:24:57.434495       1 cfg.go:46] PUSH_DOCKERCFG_PATH=/var/run/secrets/openshift.io/push/.dockercfg
I0722 03:24:57.434501       1 cfg.go:78] Found Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I0722 03:24:57.434799       1 cfg.go:64] Using serviceaccount user for Docker authentication
I0722 03:24:57.434841       1 sti.go:145] Using provided push secret for pushing 172.30.234.245:5000/chunp/origin-ruby-sample:latest image
I0722 03:24:57.434850       1 sti.go:148] Pushing 172.30.234.245:5000/chunp/origin-ruby-sample:latest image ...
Image successfully pushed
Digest: sha256:f8699168b06d66cc6b9bb03cfccdbdfdba2ef93cbc8e7f49dd554c85034fdd0a
I0722 03:26:56.495641       1 sti.go:152] Successfully pushed 172.30.234.245:5000/chunp/origin-ruby-sample:latest

Comment 5 Cesar Wong 2015-07-24 00:38:01 UTC
This is working for me on Fedora but not on RHEL. RHEL is ignoring the --confirm-def-push=false flag. And there's a bug open for that:

https://bugzilla.redhat.com/show_bug.cgi?id=1241952

Comment 6 Cesar Wong 2015-07-24 18:56:08 UTC
The workaround for this is to edit /var/lib/docker/repositories-devicemapper:

Change the value of "ConfirmDefPush" from true to false.

Restart the Docker daemon.

It is fixed in docker 1.7 for RHEL which will be released shortly. Given the workaround and that it will be fixed shortly, I'm lowering the severity.

Comment 7 chunchen 2015-09-09 05:43:17 UTC
The issue is still reproduced when docker is 1.7.1 and the value of "ConfirmDefPush" is true

[root@openshift-138 ~]# docker version
Client version: 1.7.1
Client API version: 1.19
Package Version (client): docker-1.7.1-108.el7.x86_64
Go version (client): go1.4.2
Git commit (client): 3043001/1.7.1
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Package Version (server): docker-1.7.1-108.el7.x86_64
Go version (server): go1.4.2
Git commit (server): 3043001/1.7.1
OS/Arch (server): linux/amd64

Comment 8 Andy Goldstein 2015-09-09 10:00:05 UTC
--confirm-def-push is a custom addition to the RHEL Docker RPM. There is no ability currently for us to confirm that it's ok to push to the Hub in the Docker client API that we use when pushing after a completed build. I'm not sure there's anything we can do here.

Comment 9 Cesar Wong 2015-09-09 13:54:14 UTC
Andy, do we document somewhere that the --confirm-def-push needs to be turned off in order for push to docker hub to work?

Comment 10 Andy Goldstein 2015-09-09 16:49:40 UTC
Cesar - I don't know offhand.

Comment 11 Cesar Wong 2015-10-23 10:54:35 UTC
chunchen looking at the history of this bug, not sure if there is really a bug anymore. The version of docker we have in RHEL requires you to have the --confirm-def-push=false flag set on the daemon so that we can push to the DockerHub. At some point that flag was broken and you had to manually edit the repositories-devicemapper file to get the same effect. That part of it should be fixed, but you still need to specify the flag.

Comment 12 Ben Parees 2015-10-26 20:33:02 UTC
chunchen can you please try again with the --confirm-def-push=false flag set on your host's docker daemon?

thanks.

Comment 13 chunchen 2015-10-27 11:04:19 UTC
It works well with the --confirm-def-push=false flag set on docker daemon.

I1027 06:55:55.713080       1 sti.go:296] Successfully built docker.io/chunyunchen/origin-ruby-sample-sti2:latest
I1027 06:56:04.547556       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build531069471
I1027 06:56:04.555606       1 fs.go:99] Removing directory '/tmp/s2i-build531069471'
I1027 06:56:04.566829       1 sti.go:210] Using provided push secret for pushing docker.io/chunyunchen/origin-ruby-sample-sti2:latest image
I1027 06:56:04.566852       1 sti.go:214] Pushing docker.io/chunyunchen/origin-ruby-sample-sti2:latest image ...
I1027 07:00:27.301472       1 sti.go:230] Successfully pushed docker.io/chunyunchen/origin-ruby-sample-sti2:latest

Comment 14 Brenton Leanhardt 2015-11-23 14:24:24 UTC
This fix is available in OpenShift Enterprise 3.1.


Note You need to log in before you can comment on or make changes to this bug.