Bug 1245524 - Failed to start build after setting the push secret
Summary: Failed to start build after setting the push secret
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.0.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: ---
Assignee: Cesar Wong
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On: 1232694
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-22 08:47 UTC by Miheer Salunke
Modified: 2019-08-15 04:55 UTC (History)
7 users (show)

Fixed In Version: openshift-3.0.1.0-1.git.388.51e9da7.el7ose
Doc Type: Bug Fix
Doc Text:
Clone Of: 1232694
Environment:
Last Closed: 2015-09-08 17:57:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Miheer Salunke 2015-07-22 08:47:27 UTC
+++ This bug was initially created as a clone of Bug #1232694 +++

Description of problem:
It's failed to start build after setting the push secret, met "Error creating: Pod "ruby-sample-build-1-build" is invalid: spec.volumes[2].name: duplicate value 'pushme'".

Version-Release number of selected component (if applicable):
devenv-fedora_1822 (ami-a74ebdcc)

How reproducible:
always

Steps to Reproduce:
1. Setup the OpenShift V3 environment according to: OpenShift 3 Application Lifecycle Sample  and login to server
   $ oc login --server=$master -u $user
   $ oc new-project $project-name

2. Get the '.dockercfg' file which contain account credential
1) register an account on DockerHub
2) log in to the registry to get the account credential
   $ docker login ## the credential file named ".dockercfg" will be generated under $HOME directory after login successfully

3. Create the secret
   $ oc secrets new pushme ~/.dockercfg

4. Update service account "builder" and secret "pushme" to make sure service account reference the secret
   $ oc get serviceaccount
   $ oc get secret
   $ oc edit serviceaccount/builder   # add secret "pushme" under "secrets" section as below
<---------snip-------->
secrets:
- name: pushme

   $ oc edit secret/pushme # add service account "builder" under "metadata" section as below
<---------snip-------->
metadata:
  annotations:
    kubernetes.io/service-account.name: builder
    kubernetes.io/service-account.uid: <builder-service-account-uid>

5. Update template json file to set the push secret like below
$ cat application-template-stibuild.json
     <----------snip--------->
       {
         "kind": "ImageStream",
         "apiVersion": "v1beta3",
         "metadata": {
           "name": "origin-ruby-sample"
         },
         "spec":{
         "dockerImageRepository": "chunyunchen/origin-ruby-sample"  ## my Dockerhub registry
       },
             "parameters": {
        "output": {
          "to": {
            "name": "origin-ruby-sample"
          },
          "pushSecret":{
             "name":"pushme"   ## the "pushSecret.name" is the same with above secret name
          }
        },
     <----------snip--------->
        "strategy": {
          "type": "Source"
        }
      },
     <----------snip--------->

6. Create the application with above template file
   $ oc process -f application-template-stibuild.json | oc create -f -

7. Check the build
   $ oc get build
   $ oc describe build/ruby-sample-build-1

Actual results:
the build's status is always "New" due to the pull secret is set automatically with same name of push secret, like below:

Name:			ruby-sample-build-1
Created:		9 minutes ago
Labels:			buildconfig=ruby-sample-build,name=ruby-sample-build,template=application-template-stibuild
Build Config:		ruby-sample-build
Status:			New
Duration:		waiting for 16m56s
Build Pod:		ruby-sample-build-1-build
Strategy:		Source
Image Reference:	DockerImage openshift/ruby-20-centos7:latest
Pull Secret Name:	pushme
Incremental Build:	yes
Source Type:		Git
URL:			git://github.com/openshift/ruby-hello-world.git
Output to:		origin-ruby-sample:latest
Output Spec:		<none>
Push Secret:		pushme
Events:
  FirstSeen				LastSeen			Count	From			SubobjectPath	Reason		Message
  Wed, 17 Jun 2015 17:24:29 +0800	Wed, 17 Jun 2015 17:33:50 +0800	6	{build-controller }			failedCreate	Error creating: Pod "ruby-sample-build-1-build" is invalid: spec.volumes[2].name: duplicate value 'pushme'

Expected results:
Should start build after setting the push secret successfully.

Additional info:

--- Additional comment from Michal Fojtik on 2015-06-17 09:21:05 EDT ---

chunchen: The workaround for this is simple, just don't use the same secret name for both push/pull secrets. We have to make the volume names more unique to allow this.

--- Additional comment from chunchen on 2015-06-18 04:12:57 EDT ---

@Michal Fojtik

On OSE, also meet "API error (403)" when pushing image to Dockerhub registry even have added option "--confirm-def-push=false" to docker instance like below:

root      14332      1  4 14:44 ?        00:00:15 /usr/bin/docker -d --selinux-enabled --confirm-def-push=false -b=lbr0 --mtu=1450 --insecure-registry 0.0.0.0/0
root      14821  14332  0 14:44 ?        00:00:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 1936 -container-ip 10.1.0.6 -container-port 1936
root      14833  14332  0 14:44 ?        00:00:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 10.1.0.6 -container-port 443
root      14845  14332  0 14:44 ?        00:00:00 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 10.1.0.6 -container-port 80
1000000+  15309  14332  0 14:44 ?        00:00:00 /usr/bin/dockerregistry /config.yml

Some logs:
<------------snip---------->
I0618 04:00:02.442869       1 sti.go:246] Successfully built chunyunchen/ose-sti-0618
I0618 04:00:03.817883       1 cleanup.go:23] Removing temporary directory /tmp/sti066613609
I0618 04:00:03.817935       1 fs.go:99] Removing directory '/tmp/sti066613609'
I0618 04:00:03.821960       1 cfg.go:78] Found Docker authentication configuration in '/root/.dockercfg'
I0618 04:00:03.821988       1 cfg.go:46] PUSH_DOCKERCFG_PATH=/var/run/secrets/openshift.io/push/.dockercfg
I0618 04:00:03.822000       1 cfg.go:78] Found Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I0618 04:00:03.822217       1 cfg.go:64] Using chunyunchen user for Docker authentication
I0618 04:00:03.822235       1 sti.go:96] Using provided push secret for pushing chunyunchen/ose-sti-0618 image
I0618 04:00:03.822248       1 sti.go:99] Pushing chunyunchen/ose-sti-0618 image ...
E0618 04:00:03.824341       1 dockerutil.go:50] push for image chunyunchen/ose-sti-0618 failed, will retry in 10s ...
E0618 04:00:13.826163       1 dockerutil.go:50] push for image chunyunchen/ose-sti-0618 failed, will retry in 10s ...
F0618 04:00:24.818410       1 builder.go:70] Build error: Failed to push image: API error (403): Error: Status 403 trying to push repository chunyunchen/ose-sti-0618 to official registry: needs to be forced

--- Additional comment from chunchen on 2015-06-18 06:02:38 EDT ---

Please ignore Comment #2, will try again when bug [1] is fixed, sorry for confused.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1224210

--- Additional comment from openshift-github-bot on 2015-06-26 13:50:39 EDT ---

Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/01afcf9008e319321e96a36a02f3e8334e11a154
Bug 1232694 - Make the secret volume for push/pull secrets unique

--- Additional comment from chunchen on 2015-07-03 02:27:03 EDT ---

It's fixed, verified on devenv_fedora_1921, please refer to the below results:

[chunchen@F17-CCY cmd]$ oc describe build/ruby-sample-build-1
Name:			ruby-sample-build-1
Created:		Less than a second ago
Labels:			buildconfig=ruby-sample-build,name=ruby-sample-build,template=application-template-stibuild
Build Config:		ruby-sample-build
Status:			Running
Started:		2015-07-03 13:49:07 +0800 CST
Duration:		running for 24s
Build Pod:		ruby-sample-build-1-build
Strategy:		Source
Image Reference:	DockerImage openshift/ruby-20-centos7:latest
Pull Secret Name:	pushme
Source Type:		Git
URL:			https://github.com/openshift/ruby-hello-world.git
Output to:		origin-ruby-sample:latest
Output Spec:		<none>
Push Secret:		pushme
Events:
  FirstSeen				LastSeen			Count	From				SubobjectPath				Reason		Message
  Fri, 03 Jul 2015 13:49:04 +0800	Fri, 03 Jul 2015 13:49:04 +0800	1	{scheduler }								scheduled	Successfully assigned ruby-sample-build-1-build to ip-10-150-123-20
  Fri, 03 Jul 2015 13:49:04 +0800	Fri, 03 Jul 2015 13:49:04 +0800	1	{kubelet ip-10-150-123-20}	implicitly required container POD	pulled		Successfully pulled image "openshift/origin-pod:latest"
  Fri, 03 Jul 2015 13:49:05 +0800	Fri, 03 Jul 2015 13:49:05 +0800	1	{kubelet ip-10-150-123-20}	implicitly required container POD	created		Created with docker id 093ed00ab9b0f47b1863316820a22f49547b6420ad4ba928b4b09850ec02c251
  Fri, 03 Jul 2015 13:49:05 +0800	Fri, 03 Jul 2015 13:49:05 +0800	1	{kubelet ip-10-150-123-20}	implicitly required container POD	started		Started with docker id 093ed00ab9b0f47b1863316820a22f49547b6420ad4ba928b4b09850ec02c251
  Fri, 03 Jul 2015 13:49:06 +0800	Fri, 03 Jul 2015 13:49:06 +0800	1	{kubelet ip-10-150-123-20}	spec.containers{sti-build}		created		Created with docker id 7e8010c8a7feedc1742f0b4509b6f703a4d7f5490a111d6ae95b60aa1c1c781d
  Fri, 03 Jul 2015 13:49:06 +0800	Fri, 03 Jul 2015 13:49:06 +0800	1	{kubelet ip-10-150-123-20}	spec.containers{sti-build}		started		Started with docker id 7e8010c8a7feedc1742f0b4509b6f703a4d7f5490a111d6ae95b60aa1c1c781d

Comment 5 Gaoyun Pei 2015-07-23 07:20:45 UTC
Verify this bug with openshift-3.0.1.0-1.git.388.51e9da7.el7ose.x86_64

After adding secret "pushme" to builder service account, trigger a new sti-build, the build could get started.

[gpei@master ~]$ oc describe build ruby-sample-build-3
Name:			ruby-sample-build-3
Created:		23 seconds ago
Labels:			buildconfig=ruby-sample-build,name=ruby-sample-build,template=application-template-stibuild
Build Config:		ruby-sample-build
Status:			Running
Started:		2015-07-23 14:45:10 +0800 CST
Duration:		running for 20s
Build Pod:		ruby-sample-build-3-build
Strategy:		Source
Image Reference:	DockerImage openshift/ruby-20-centos7:latest
Pull Secret Name:	pushme
Source Type:		Git
URL:			https://github.com/openshift/ruby-hello-world.git
Output to:		ImageStreamTag origin-ruby-sample:latest
Push Secret:		pushme
Events:
  FirstSeen				LastSeen			Count	From				SubobjectPath			Reason		Message
  Thu, 23 Jul 2015 14:45:08 +0800	Thu, 23 Jul 2015 14:45:08 +0800	1	{scheduler }							scheduled	Successfully assigned ruby-sample-build-3-build to node.cluster.local
  Thu, 23 Jul 2015 14:45:08 +0800	Thu, 23 Jul 2015 14:45:08 +0800	1	{kubelet node.cluster.local}	implicitly required container PODpulled		Pod container image "registry.access.redhat.com/openshift3/ose-pod:v3.0.1.0" already present on machine
  Thu, 23 Jul 2015 14:45:09 +0800	Thu, 23 Jul 2015 14:45:09 +0800	1	{kubelet node.cluster.local}	implicitly required container PODcreated		Created with docker id ce940bb7a833
  Thu, 23 Jul 2015 14:45:09 +0800	Thu, 23 Jul 2015 14:45:09 +0800	1	{kubelet node.cluster.local}	implicitly required container PODstarted		Started with docker id ce940bb7a833
  Thu, 23 Jul 2015 14:45:10 +0800	Thu, 23 Jul 2015 14:45:10 +0800	1	{kubelet node.cluster.local}	spec.containers{sti-build}	created		Created with docker id a4ca1e23f4e9
  Thu, 23 Jul 2015 14:45:10 +0800	Thu, 23 Jul 2015 14:45:10 +0800	1	{kubelet node.cluster.local}	spec.containers{sti-build}	started		Started with docker id a4ca1e23f4e9

Comment 6 Josep 'Pep' Turro Mauri 2015-09-08 17:57:17 UTC
This issue was addressed in OpenShift Enterprise 3.0.1.0:

https://bugzilla.redhat.com/show_bug.cgi?id=1232694


Note You need to log in before you can comment on or make changes to this bug.