Bug 1268478 - docker builder cannot retrieve source from git when user name is not a private git repository.
Summary: docker builder cannot retrieve source from git when user name is not a priva...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Gabe Montero
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks: 1268936
TreeView+ depends on / blocked
 
Reported: 2015-10-02 21:23 UTC by Ryan Howe
Modified: 2022-07-09 07:37 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1268936 (view as bug list)
Environment:
Last Closed: 2016-01-26 19:16:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:0070 0 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 3.1.1 bug fix and enhancement update 2016-01-27 00:12:41 UTC

Description Ryan Howe 2015-10-02 21:23:16 UTC
Description of problem:
docker builder cannot retrieve source from git when user name  is not git

Version-Release number of selected component (if applicable):
3.0.2

How reproducible:
Customer

Steps to Reproduce:
Create a build config with you git source of you private repo. 

source:
    git:
      uri: mbalazs.hu:/var/git/eap-ulx.git


Actual results:
F0924 11:04:03.821101       1 builder.go:54] Build error: dial tcp: missing address

Expected results:
Builds the image 

Additional info:
This could be a docs bug as well as the docs give this:
"The URL of private repository is usually in the form git:<username>/<repository>."

Comment 2 Ben Parees 2015-10-05 12:07:02 UTC
the proper way to provide auth credentials is via a secret now:

https://docs.openshift.org/latest/dev_guide/builds.html#basic-authentication

Comment 3 Ryan Howe 2015-10-06 18:03:45 UTC
This bug is when a user is used that is not named git to ssh in. We provided auth credentials via a secret.  This may be a docs bug, but do we need to use git as the user for a private repo?   



F1006 13:56:46.390068       1 builder.go:54] Build error: dial tcp: missing address


[master ~]$ oc get bc -o yaml  gitbuild
apiVersion: v1
kind: BuildConfig
metadata:
  creationTimestamp: 2015-10-06T15:41:23Z
  name: gitbuild
  namespace: fun
  resourceVersion: "1243431"
  selfLink: /oapi/v1/namespaces/fun/buildconfigs/gitbuild
  uid: b26e5877-6c40-11e5-80e3-fa163e7a4b21
spec:
  output:
    to:
      kind: DockerImage
      name: registry-route.default.cloudapps.lizard.com/fun/test
  resources: {}
  source:
    git:
      uri: root.com:root/project.git
    sourceSecret:
      name: scmsecret
    type: Git
  strategy:
    dockerStrategy: {}
    type: Docker
  triggers: []
status:
  lastVersion: 4

Comment 4 Cesar Wong 2015-10-07 18:05:12 UTC
There is a bug in new-app and the builders in the handling of ssh URLs (with a user that is not 'git'). I am going to assign this bug to myself and deliver a fix for it.

Comment 5 Gabe Montero 2015-10-23 13:17:15 UTC
As a point of ref, the git doc on the uri format possibilities:  https://git-scm.com/book/ch4-1.html

Comment 6 Gabe Montero 2015-10-28 15:25:08 UTC
Update: fix is in the PR pipeline.

Once https://github.com/openshift/source-to-image/pull/319 gets merged, a godeps update and analogous PR for origin will commence.

Comment 7 Gabe Montero 2015-11-02 19:57:02 UTC
The s2i pull has been merged, and the origin side changes are getting reviewed under https://github.com/openshift/origin/pull/5594

Comment 8 Gabe Montero 2015-11-16 15:45:18 UTC
FYI - the fixes merged into https://github.com/openshift/origin this past Friday with pull https://github.com/openshift/origin/pull/5863

Moving this item onto QA - please monitor the AEP/OSE puddles for a version that picks up the above referenced fix.

thanks

Comment 9 XiuJuan Wang 2015-11-17 08:21:37 UTC
Check in devenv-rhel7_2717 instance. Code has merged.

Still could reproduce this bug.

spec:
  output:
    to:
      kind: ImageStreamTag
      name: origin-ruby-sample:latest
  resources: {}
  source:
    git:
      uri: xiuwang:xiuwang/ruby-hello-world.git
    sourceSecret:
      name: mysecret

$oc  build-logs ruby-sample-build-1 
F1117 08:17:32.182471       1 builder.go:59] Build error: dial tcp: missing address

Comment 10 Gabe Montero 2015-11-17 17:04:10 UTC
OK ... i've reproduced locally ... there are additional bugs beyond all the clone spec parsing previously addressed.

Working on fix.

Comment 11 Gabe Montero 2015-11-17 22:32:07 UTC
Still digging through the layers, but wanted to provide an update, as well as get a clarification:

First, the clarification:  just wanted to make sure that the secret provided was a ssh-auth / ssh-privatekey, and that it was *NOT* a user/password basic auth.  While username/password is a possible way of authenticating with ssh, it requires extensions, utilities, methods the OpenShift does not support.

Please update the bug to confirm when the opportunity arises.

Next, problems uncovered:

1) a common theme we say with the git clone spec validation ... golang's URL code is not a perfect vehicle for git clone specs for the ssh protocol;  there is a problem with using URL.String() to set the Git.URI field

2) another golang item ... the net.DialTCP function used by the crypto packages does not handle plugging in the default ports for the ssh protocol ... if the port is not specified, we have to add it

I'm now at least connecting to the repository, and am sorting through the handling of ssh keys next.

Comment 12 Gabe Montero 2015-11-19 20:20:25 UTC
Once pull https://github.com/openshift/origin/pull/5956 is merged and subsequently incorporated in a devenv*, etc., the fix should be ready for verification.

Note: the scenario noted in the origination text, and git clone to a private repo should be fine, however for hosted git servers like github.com or bitbucket.org (like the attempted documented by xiuwang on 2015-11-17) will not work for users other than "git".  Please see https://help.github.com/articles/error-permission-denied-publickey/ for reference.  I confirmed through experimentation and discussion with Cesar.

Comment 13 openshift-github-bot 2015-11-21 05:39:49 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/f44f4fb0423451835a7f55f13003cf8ee4a228ac
Merge pull request #5956 from gabemontero/bug1268478-postParse

Merged by openshift-bot

Comment 14 Gabe Montero 2015-11-23 02:24:47 UTC
The fix has merged.

While verifying, please note my update in Comment 12 about when you can use non-git servers and when you can't.

Comment 15 XiuJuan Wang 2015-11-23 10:45:15 UTC
@Gabe,
In the comment #12, you said, can't use bitbacker.org and github.com with username, so I create a private git server, and create a secret using private-sshkey, but still get "Build error: dial tcp: missing"

I setup my git server following http://tumblr.intranation.com/post/766290565/how-set-up-your-own-private-git-server-linux in the devenv-rhel7-2778 instance.
And I can git clone my repo xiuwang@localhost:myrepo.git in the instance.

$oc get bc 
  spec:
    output:
      to:
        kind: ImageStreamTag
        name: origin-ruby-sample:latest
    resources: {}
    source:
      git:
        uri: xiuwang@localhost:myrepo.git
      sourceSecret:
        name: mysecret
      type: Git

If I misunderstand your comment?

Comment 16 Gabe Montero 2015-11-23 14:23:17 UTC
Based on the error, it appears you are not at a level with the fix.  Part of the fix replaced the use of golang's DialTCP function.  The error comes from the golang DialTCP function.

Next, "localhost" won't resolve in the builder container to the git server like you think it will.  You need to put the explicit hostname/IP address of the server you are trying to contact.

So first, confirm you are using a build level that has the commit level for the Confirm you are using a build with the pull https://github.com/openshift/origin/pull/5956.

Then, fix the git uri in your gc to have an explicit IP address.

Finally, if you still have problems, re-run with launching the openshift server with --loglevel=4, and gather the build container logs and I'll take a look.

thanks,
gabe

Comment 17 Gabe Montero 2015-11-23 14:47:44 UTC
Sorry - a couple of typo's in that last comment.

Let me try to fix a couple of the bullets:

- the error message tells us golang's DialTCP is still being used; this usage was removed with the fix; confirm you are at a instance that has the pull https://github.com/openshift/origin/pull/5956

- fix the git uri in your bc (not gc ;-) ); localhost host in the builder container won't resolve to the IP your git server is set up on (I made the same mistake in my testing).

Comment 18 XiuJuan Wang 2015-11-24 10:27:30 UTC
Verified in devenv-rhel7-2782

1.Create a private git server in the same instance and create a private repo xiuwang.9.188:ruby-hello-world.git.

2.Create a secret with the private-sshkey

3.Use the private repo as source repo and reffer to the sshkey secret.

 resources: {}
  source:
    git:
      uri: xiuwang.9.188:ruby-hello-world.git
    sourceSecret:
      name: mysecret
    type: Git

4.Create app and trigger build.

5.Check build.

Result: Could sti build with username(not git) of a private repo

@Gabe, sorry, when I launched openshift server yesterday, I forgot to add --images='openshift/origin-${component}:latest paramter, so I used the old origin-sti-build image.
I use the latest origin-sti-build image today, this bug has been fixed.
Thank!

Comment 19 Gabe Montero 2015-11-24 14:19:35 UTC
@XiuJuan - glad it worked out - thanks for you persistence and patience !

Comment 20 XiuJuan Wang 2015-11-25 02:09:48 UTC
Notice this bug reported against openshift enterprise,so move the bug status to on_qa back, then will verify again when the code push to ose env.

Comment 21 XiuJuan Wang 2015-12-09 03:29:54 UTC
Still don't fix in latest ose env.Will keep an eye on it.
$oc version
oc v3.1.0.4-16-g112fcc4
kubernetes v1.1.0-origin-1107-g4c8e6f4

Comment 22 XiuJuan Wang 2015-12-28 03:31:36 UTC
Fixed in ose env
oc v3.1.1.0
kubernetes v1.1.0-origin-1107-g4c8e6f4

Steps are same with comment#18 steps, move this bug to verified.

Comment 24 errata-xmlrpc 2016-01-26 19:16:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070


Note You need to log in before you can comment on or make changes to this bug.