Bug 1228189 - Failed to recognize secret when build with private source repo and build from private docker image
Summary: Failed to recognize secret when build with private source repo and build from...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.0.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Michal Fojtik
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-04 11:33 UTC by DeShuai Ma
Modified: 2015-09-08 17:35 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-08 17:35:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1229319 0 unspecified CLOSED Docker in RHEL should not claim /run/secrets directory 2021-02-22 00:41:40 UTC

Internal Links: 1229319

Description DeShuai Ma 2015-06-04 11:33:08 UTC
Description of problem:
When in ose env do build from private git repo, it always tip Cannot setup secret file for accessing private repository.

Version-Release number of selected component (if applicable):
openshift v0.5.2.2-22-g84b1674-dirty
kubernetes v0.17.1-804-g496be63

How reproducible:
Always

Steps to Reproduce:
1. Generate a ssh key and upload the public key to github
$ ssh-keygen
$ cat ~/.ssh/id_rsa.pub

2. Create a new project
$ osadm new-project test

3. Create a secret
$ cat secret.json
{
"apiVersion": "v1beta3",
"kind": "Secret",
"metadata": {
 "name": "mysecret"
},
"data": {
 "ssh-privatekey": "<<< place here the result of base64 -w 0 ~/.ssh/id_rsa >>>"
}
}
$ osc create -f secret.json -n test

4. Edit application-template-stibuild.json, add ref info to it like below:
$ cd /data/src/github.com/openshift/origin/examples/sample-app
$ vim application-template-stibuild.json
{
   "apiVersion": "v1beta1",
   "kind": "BuildConfig",
   "metadata": {
     "name": "ruby-sample-build",
...
     "source": {
       "git": {
         "uri": "git:openshift/ruby-hello-world.git"
       },
        "SourceSecret":  { "Name": "mysecret" },
       "type": "Git"
     },
}

5. Submit the application template for processing and create the application using the processed template:
$ osc process -n test -f application-template-stibuild.json | osc create -n test -f -

6. Start a build and check the build result
$ osc start-build $buildConfig -n test
$ osc get build -n test

Actual results:
6.The build failed.
[root@dhcp-128-7 examples]# osc get build
NAME                  TYPE      STATUS    POD
ruby-sample-build-1   Source    Failed    ruby-sample-build-1-build

[root@dhcp-128-7 examples]# osc build-logs ruby-sample-build-1 
I0604 05:28:48.554216       1 cfg.go:50] Problem accessing /root/.dockercfg: stat /root/.dockercfg: no such file or directory
F0604 05:28:48.572011       1 builder.go:70] Cannot setup secret file for accessing private repository: open /var/run/secrets/source: no such file or directory

Expected results:
6.The build should complete

Additional info:
http://fpaste.org/228757/41114514/
1) When enable selinux on the node, run docker directly and mount a directory to container, inside the container, it always Permission denied.
2) Disable selinux, run container with mounting a directory it works well, but when mount secret(/var/lib/openshift/openshift.local.volumes/pods/1970b98d-0a9c-11e5-8d20-fa163e3a9960/volumes/kubernetes.io~secret/mysecret), inside container it always tip no such file or directory

Comment 1 DeShuai Ma 2015-06-04 11:35:14 UTC
when use secret to build with private docker repo, also meet same issue.
[wzheng@rh6-openshift origin]$ osc build-logs ruby-sample-build-3
I0604 06:45:39.955428       1 cfg.go:46] PULL_DOCKERCFG_PATH=/var/run/secrets/pull/.dockercfg
I0604 06:45:39.974687       1 cfg.go:50] Problem accessing /var/run/secrets/pull/.dockercfg: stat /var/run/secrets/pull/.dockercfg: no such file or directory
I0604 06:45:39.974743       1 sti.go:67] Creating a new S2I builder with build request: api.Request{BaseImage:"docker.io/wzheng/ruby-20-centos7:latest", DockerConfig:(*api.DockerConfig)(0xc208043380), DockerCfgPath:"/var/run/secrets/pull/.dockercfg", PullAuthentication:docker.AuthConfiguration{Username:"", Password:"", Email:"", ServerAddress:""}, PreserveWorkingDir:false, Source:"git://github.com/wzheng1/ruby-hello-world.git", Ref:"", Tag:"172.30.33.217:5000/test2/origin-ruby-sample:latest", Incremental:true, RemovePreviousImage:false, Environment:map[string]string{"OPENSHIFT_BUILD_NAMESPACE":"test2", "OPENSHIFT_BUILD_SOURCE":"git://github.com/wzheng1/ruby-hello-world.git", "OPENSHIFT_BUILD_NAME":"ruby-sample-build-3"}, CallbackURL:"", ScriptsURL:"", Location:"", ForcePull:false, WorkingDir:"", LayeredBuild:false, InstallDestination:"", Quiet:false, ContextDir:""}
I0604 06:45:39.976125       1 docker.go:176] Pulling image docker.io/wzheng/ruby-20-centos7:latest
I0604 06:45:41.712258       1 docker.go:179] An error was received from the PullImage call: Error: image wzheng/ruby-20-centos7:latest not found
F0604 06:45:41.712302       1 builder.go:75] Build error: unable to get docker.io/wzheng/ruby-20-centos7:latest

Comment 3 Michal Fojtik 2015-06-04 18:41:40 UTC
DeShuai Ma can you please re-test this on latest master?

Comment 4 DeShuai Ma 2015-06-05 04:31:00 UTC
(In reply to Michal Fojtik from comment #3)
> DeShuai Ma can you please re-test this on latest master?

Test on latest ose env, it still have the same issue.In ec2 all-in-one env don't have this issue.
openshift version:
openshift v0.5.2.2-26-g701be15
kubernetes v0.17.1-804-g496be63

Comment 5 Michal Fojtik 2015-06-05 07:58:56 UTC
Since this is working fine with latest origin master, I think the issue is that OSE does not have commits that make this works (yet).

Steve, is there a plan to rebase for OSE soon? I would want to retest with rebased code to see if the issue if fixed.

Comment 6 Brenton Leanhardt 2015-06-05 12:44:12 UTC
This bug is assigned against Enterprise.  If we want to test it there please include a pull request so know when to move it to ON_QA once it's built.  If it was urgent we'd cherry-pick it in to beta4.

If we want QE to test against the latest master technically the process would be to clone this bug to Origin.

Comment 7 Michal Fojtik 2015-06-05 13:15:25 UTC
Brenton, thanks! I will go investigate what commits are missing in OSE to make this works. All this was reworked in last week.

Comment 8 Michal Fojtik 2015-06-05 13:19:56 UTC
DeShuai Ma can you please provide the image versions as well? Namely sti-builder (docker images | grep sti)

Comment 9 DeShuai Ma 2015-06-05 14:23:48 UTC
(In reply to Michal Fojtik from comment #8)
> DeShuai Ma can you please provide the image versions as well? Namely
> sti-builder (docker images | grep sti)

1)Enable selinux in the env.
2)In the Additional info we can see sti version is: registry.access.redhat.com/openshift3_beta/ose-sti-builder:v0.5.2.2
3)I thinks it's a common issue when mount secret to container, In ose env when mount common file or directory, it works well. But secret file is special,can't find inside container:
"/var/run/secrets/kubernetes.io/serviceaccount": "/var/lib/openshift/openshift.local.volumes/pods/1970b98d-0a9c-11e5-8d20-fa163e3a9960/volumes/kubernetes.io~secret/default-token-y65hg",
        "/var/run/secrets/source": "/var/lib/openshift/openshift.local.volumes/pods/1970b98d-0a9c-11e5-8d20-fa163e3a9960/volumes/kubernetes.io~secret/mysecret"

Comment 16 Daniel Walsh 2015-06-08 13:07:38 UTC
If we mount this directory before other volume mounts it should fix the problem correct?

The problem with changing the directory now, would be an upgrade problem with existing images.  I believe they look directly at this directory.

Comment 17 Michal Fojtik 2015-06-08 13:16:45 UTC
(In reply to Daniel Walsh from comment #16)
> If we mount this directory before other volume mounts it should fix the
> problem correct?
> 
> The problem with changing the directory now, would be an upgrade problem
> with existing images.  I believe they look directly at this directory.

Yes, if we mount RHEL /run/secrets before other mounts that might fix it. Right, fixing this problem by namespacing the dir will require to fix the symlink to /etc/yum... i guess... which will require to rebuild all images.

Comment 18 Daniel Walsh 2015-06-08 14:07:46 UTC
And break all existing images.  Not a good situation.  I have just pushed a fix to docker-1.6.2 git hub for rhel.  I will see about getting a build out today.

Comment 19 Michal Fojtik 2015-06-08 14:25:05 UTC
DeShuai can you please re-test when the updated Docker RPM will be available? 

See:https://bugzilla.redhat.com/show_bug.cgi?id=1229319

Comment 23 Michal Fojtik 2015-06-08 17:47:17 UTC
The Source secrets are testable in OSE (this bug was originally about that).

Comment 24 DeShuai Ma 2015-06-09 01:50:30 UTC
Test on ose again and this bug is fixed.
openshift version
openshift v0.5.2.2-26-g701be15
kubernetes v0.17.1-804-g496be63
docker version : docker-1.6.2-8.el7.x86_64

Comment 25 Josep 'Pep' Turro Mauri 2015-09-08 17:35:15 UTC
Closing this as part of a bulk update/cleanup of multiple bugs that were VERIFIED before OSE 3.0 GA but were left open and haven't been updated since.

If this bug was meant to stay open for some reason please reopen.


Note You need to log in before you can comment on or make changes to this bug.