Bug 1657187

Summary: ose-docker-builder does not login to registry.redhat.io when doing builds
Product: OpenShift Container Platform Reporter: Dmitry Zhukovski <dzhukous>
Component: BuildAssignee: Ben Parees <bparees>
Status: CLOSED NOTABUG QA Contact: wewang <wewang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aabhishe, adeshpan, amurdaca, aos-bugs, bparees, gparente, jfoots, jokerman, kborup, mmccomas, mnoguera, ocasalsa, pamoedom, rbost, sgarciam, wzheng
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-13 09:37:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
my-test-build pod objects none

Description Dmitry Zhukovski 2018-12-07 11:53:40 UTC
Description of problem:
ose-docker-builder does not login to registry.redhat.io when doing builds

Version-Release number of selected component (if applicable):
OCP 3.11.43
Docker version:
 Version: 1.13.1
 API version:     1.26
 Package version: docker-1.13.1-84.git07f3374.el7.x86_64
 Go version:      go1.10.2
 Git commit:      07f3374/1.13.1
 Built:           Tue Nov  6 18:41:32 2018
 OS/Arch:         linux/amd64

How reproducible:
everytime

Steps to Reproduce:
1. Create build in a pre-defined OCP project (e.g. oc project myproject) 
2. oc create -f buildFile.yaml
3. oc start-build case-xxx-build --from-dir=.


Actual results:
The build fails with the following message:
 
Receiving source from STDIN as archive ...

Pulling image registry.redhat.io/openshift3/ose ...
error: build error: failed to pull image: Get https://registry.redhat.io/v2/openshift3/ose/manifests/latest:  unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531 
 

Expected results:
Build successfully finishes

Additional info:

Comment 12 Antonio Murdaca 2018-12-07 14:11:30 UTC
Please also provide openshift exact version

Comment 16 Antonio Murdaca 2018-12-07 14:30:17 UTC
docker pull works fine and credentials are stored on the right paths as well, more information are coming from customer/consultant but I don't think it's a docker issue, rather an openshift misconfiguration or bug maybe. Please re-assign it to me otherwise.

Comment 17 Ben Parees 2018-12-07 15:10:36 UTC
This is working as designed.

To access the new registry you either need to setup appropriate credentials like for any other private registry, or you need to use pull through against an imagestream in a namespace that has credentials setup (see the out of box openshift namespace imagestreams). 

Or continue using registry.access.redhat.com.

Node credentials are not used for build pulls. You must supply them in your namespace.

Comment 20 Robert Bost 2018-12-15 01:12:50 UTC
> To access the new registry you either need to setup appropriate credentials like for any other private registry

FTR, this would be done like below and will be entered into red hat knowledgebase. Note the `docker login` steps and `oc create secret` can be abbreviate if you want to enter credentials into `oc create secret` directly without logging in via docker (e.g. if you do not have it avialable locally and only use `oc`):

# cat imagestream-and-bc.yaml
kind: "List"
apiVersion: "v1"
items:
- kind: "ImageStream"
  apiVersion: "v1"
  metadata: 
    labels:
      app: "my-test"
      template: "my-test"
    name: "my-test"
- kind: "BuildConfig"
  apiVersion: "v1"
  metadata: 
    labels:
      template: "my-test"
    name: "my-test-build"
  spec:
    source:
      binary: {}
      type: Binary
    strategy: 
      type: Docker
      dockerStrategy:
        dockerfilePath: .
    triggers: []
    output: 
      to:
        kind: "ImageStreamTag"
        name: "my-test:latest"
# cat Dockerfile 
FROM registry.redhat.io/openshift3/ose
USER 1001
CMD ["/bin/ls"]
# oc new-project my-test
# oc create -f imagestream-and-bc.yaml
imagestream.image.openshift.io/my-test created
buildconfig.build.openshift.io/my-test-build created
# docker login registry.redhat.io 
<enter credentials>
# oc create secret generic my-credentials --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson
# oc start-build my-test-build --from-dir=.
Uploading directory "." as binary input for the build ...
.
Uploading finished
build.build.openshift.io/my-test-build-2 started
# oc logs my-test-build-2-build 
Step 1/5 : FROM registry.redhat.io/openshift3/ose
 ---> 96ee92cf05ea
Step 2/5 : USER 1001
 ---> Running in 6269bc22ba23
 ---> ebf4f076260b
...

Success from here on.

Comment 30 Oscar Casal Sanchez 2019-04-16 12:23:08 UTC
Created attachment 1555501 [details]
my-test-build pod objects

Comment 44 Sergio G. 2019-08-13 08:53:42 UTC
As an extra note, if I create an imagestream to point to the image, the build succeeds.

Basic instructions, working in Cluster A (the impossible thing which should be explained):
# cat imagestream-and-bc.yaml
kind: "List"
apiVersion: "v1"
items:
- kind: "ImageStream"
  apiVersion: "v1"
  metadata: 
    labels:
      app: "my-test"
      template: "my-test"
    name: "my-test"
- kind: "BuildConfig"
  apiVersion: "v1"
  metadata: 
    labels:
      template: "my-test"
    name: "my-test-build"
  spec:
    source:
      binary: {}
      type: Binary
    strategy: 
      type: Docker
      dockerStrategy:
        dockerfilePath: .
    triggers: []
    output: 
      to:
        kind: "ImageStreamTag"
        name: "my-test:latest"
# cat Dockerfile 
FROM registry.redhat.io/openshift3/ose
USER 1001
CMD ["/bin/ls"]
# oc new-project my-test
# oc create -f imagestream-and-bc.yaml
imagestream.image.openshift.io/my-test created
buildconfig.build.openshift.io/my-test-build created
# oc start-build my-test-build --from-dir=.
Uploading directory "." as binary input for the build ...
.
Uploading finished
build.build.openshift.io/my-test-build-2 started
# oc logs my-test-build-1-build 
Step 1/5 : FROM registry.redhat.io/openshift3/ose
 ---> 96ee92cf05ea
Step 2/5 : USER 1001
 ---> Running in 6269bc22ba23
 ---> ebf4f076260b
...
Success!


Extra step required in Cluster B. Any of them makes the build work:
- Option 1
  # oc import-image rhel7-minimal --from registry.redhat.io/rhel7-minimal --reference-policy local --confirm
- Option 2
  # oc create secret generic my-credentials --from-file=.dockerconfigjson=$HOME/.docker/config.json --type=kubernetes.io/dockerconfigjson
  # oc set build-secret --pull bc/my-test-build my-credentials


While I don't agree with the explanation of "from now on every single build or project must be tweaked to get images" as in the past it wasn't that way, the point here is to understand why Cluster A is working without such extra configuration.

Thanks in advance.

Comment 45 Sergio G. 2019-08-13 08:55:00 UTC
Correction to the previous update:

# cat Dockerfile 
FROM registry.redhat.io/rhel7-minimal
USER 1001
CMD ["/bin/ls"]

Comment 46 Sergio G. 2019-08-13 09:37:06 UTC
I regret the noise, but it was working because the image was already pulled in the only single node in the cluster... my bad. Closing it again.