Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1689848

Summary: example template image url does not work on Satellite
Product: OpenShift Container Platform Reporter: Daein Park <dapark>
Component: InstallerAssignee: Joseph Callen <jcallen>
Installer sub component: openshift-ansible QA Contact: Johnny Liu <jialiu>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bleanhar, jack.ottofaro, jcallen, knakai, mfuruya, mori, openshift-bugs-escalate
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.11.104-1.git.0.379a011.el7 Doc Type: Bug Fix
Doc Text:
Cause: Installer did not replace image url of example resources with the Satellite registry one. Consequence: If you install using Satellite registry, example resources are configured with invalid image urls. Fix: Added a condition to replace image url with Satellite one. Result: The example resources are configured with valid image url.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-06 02:00:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daein Park 2019-03-18 09:29:21 UTC 
Description of problem:

The all template image urls are installed as wrong path, even though oreg_rule is configured during disconnected installing using Satellite.

openshift_example role does not replace image url properly when "oreg_url" is consist with "<hostname>/<image name>".
The tasks replace only hostname part from template image url. It does not consider the <hostname>/<image name>  format. refer [0] for more details.

~~~
  $ oc describe is -n openshift jboss-webserver30-tomcat7-openshift
  Name:                   jboss-webserver30-tomcat7-openshift
  Namespace:              openshift
  ...
  1.3

   tagged from test.example.com:5000/jboss-webserver-3/webserver30-tomcat7-openshift:1.3

     prefer registry pullthrough when referencing this tag
  ...
  ! error: Import failed (InternalError): Internal error occurred: unknown: Not Found
~~~

Satellite usually uses "<hostname>/<imagename>" url for container registry, as following manner. It's required on the disconnected installation using Satellite.

  <HOSTNAME>/<ORGANIZATION>-<PRODUCT>-<REPOSITORY>
  
  For instance, if "registry.redhat.com/rhel7/etcd" image publish on the Satellite (register to Satellite as "rhel7/etcd" ),
  then the URL will be changed to "satellite.example.com/<ORGANIZATION>-<PRODUCT>-rhel7_etcd" format, and "/" is replaced with "_" on the Satellite.

  As for '"/" is replaced with "_" on the Satellite.' part, refer [1] for more details.


[0] https://github.com/openshift/openshift-ansible/blob/release-3.11/roles/openshift_examples/tasks/stream_secrets.yml#L33
~~~
    find {{ examples_base }} -type f | xargs -n 1 sed -i 's|registry.redhat.io|{{ registry_host | quote }}|g'
~~~

[1] https://github.com/Katello/bastion/blob/master/app/assets/javascripts/bastion/utils/form-utils.service.js

Version-Release number of the following components:
rpm -q openshift-ansible
openshift-ansible-3.11.82-3.git.0.9718d0a.el7.noarch

rpm -q ansible
ansible-2.6.14-1.el7ae.noarch

ansible --version
ansible 2.6.14
  config file = /usr/share/ansible/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:
Always, when you configure oreg_url=<hostname>/<imagename> using Satellite.

Steps to Reproduce:
1.
2.
3.

Actual results:
Even though you configured oreg_url=test.example.com:5000/imagename, the all template is installed as "test.example.com:5000/somepath/imagename".
It's not working.

Expected results:
All template image url has replaced properly with <hostname>/<image name> pattern on Satellite, and it works.

Additional info:
Comment 2 Daein Park 2019-03-18 13:36:49 UTC 
I've mentioned the same thing here: https://bugzilla.redhat.com/show_bug.cgi?id=1689796#c3
But this BZ is related with https://bugzilla.redhat.com/show_bug.cgi?id=1689796#c3 either, so I'm sorry for duplication, but I repeat again here.

---
Hi, 

We recommend Satellite as image registry when OCP deploy as disconnected installation as follows.
So we should provide concrete solutions for a customer who is using Satellite, such as ansible playbooks that is supported for Satellite env.
v3.11 will be supported more longer than other 3.x versions, it's important either.

* Disconnected installation - Prerequisites
  [ https://docs.openshift.com/container-platform/3.11/install/disconnected_install.html#disconnected-prerequisites ]
  ~~~
    Using a Red Hat Satellite 6.1 server that acts as a container image registry.
  ~~~
Comment 5 Joseph Callen 2019-03-22 17:32:38 UTC 
Merged https://github.com/openshift/openshift-ansible/pull/11363
Comment 14 Joseph Callen 2019-04-08 12:31:54 UTC 
In build: openshift-ansible-3.11.103-1
Comment 15 Joseph Callen 2019-04-08 12:32:27 UTC 
And previous builds: openshift-ansible-3.11.100-1, openshift-ansible-3.11.101-1, openshift-ansible-3.11.102-1
Comment 16 Johnny Liu 2019-04-09 07:59:56 UTC 
The latest available puddle is v3.11.100-1_2019-03-24.1, installer version is openshift-ansible-3.11.100-1.git.0.5a24ec5.el7.noarch in it.

Seem like the PR is not merged into build yet.
TASK [openshift_examples : Modify registry paths if registry_url is not registry.redhat.io] ***
Tuesday 09 April 2019  15:26:08 +0800 (0:00:00.067)       0:09:47.943 ********* 

changed: [vm-10-0-76-207.hosted.upshift.rdu2.redhat.com] => {"changed": true, "cmd": "find /usr/share/openshift/examples -type f | xargs -n 1 sed -i 's|registry.redhat.io|vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000|g'", "delta": "0:00:00.560315", "end": "2019-04-09 03:26:08.699123", "rc": 0, "start": "2019-04-09 03:26:08.138808", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}


Waiting for newer puddle.
Comment 19 Johnny Liu 2019-04-11 11:12:24 UTC 
Verified this bug with openshift-ansible-3.11.104-1.git.0.379a011.el7.noarch + openshift v3.11.98, and PASS.

oreg_url=vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-openshift3_ose-${component}:${version}
openshift_examples_modify_imagestreams=true
osm_etcd_image=vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-rhel7_etcd:3.2.22
openshift_docker_insecure_registries=vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000
openshift_docker_blocked_registries=registry.redhat.io

Installation log:
ASK [openshift_examples : Modify registry paths if registry_url is not registry.redhat.io] ***
Thursday 11 April 2019  15:39:37 +0800 (0:00:00.062)       0:09:40.266 ******** 
skipping: [vm-10-0-76-53.hosted.upshift.rdu2.redhat.com] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [openshift_examples : Modify registry paths if registry_url is not registry.redhat.io and using Satellite] ***
Thursday 11 April 2019  15:39:37 +0800 (0:00:00.073)       0:09:40.339 ******** 

changed: [vm-10-0-76-53.hosted.upshift.rdu2.redhat.com] => {"changed": true, "cmd": "find /usr/share/openshift/examples -type f | xargs -n 1 sed -i -e 's|registry.redhat.io/\\([^/]*\\)/\\(.*\\)$|registry.redhat.io/\\1_\\2|g' -e 's|registry.redhat.io/\\([^/]*\\)$|vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-openshift3_ose-\\1|g' -e 's/openshift3[-_]ose-//g'", "delta": "0:00:00.845966", "end": "2019-04-11 03:39:38.502099", "rc": 0, "start": "2019-04-11 03:39:37.656133", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

After installation, check example template image url.
# oc get is ruby -n openshift -o yaml
<--snip-->
  - annotations:
      description: Build and run Ruby 2.5 applications on RHEL 7. For more information
        about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/blob/master/2.5/README.md.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby 2.5
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/sclorg/ruby-ex.git
      supports: ruby:2.5,ruby
      tags: builder,ruby
      version: "2.5"
    from:
      kind: DockerImage
      name: vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-rhscl_ruby-25-rhel7:latest
    generation: 2
    importPolicy: {}
    name: "2.5"
    referencePolicy:
      type: Local
  - annotations:
      description: |-
        Build and run Ruby applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-ruby-container/tree/master/2.3/README.md.

        WARNING: By selecting this tag, your application will automatically update to use the latest version of Ruby available on OpenShift, including major versions updates.
      iconClass: icon-ruby
      openshift.io/display-name: Ruby (Latest)
      openshift.io/provider-display-name: Red Hat, Inc.
      sampleRepo: https://github.com/openshift/ruby-ex.git
      supports: ruby
      tags: builder,ruby
    from:
      kind: ImageStreamTag
      name: "2.5"
    generation: 1
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Local
<--snip-->

# oc describe is -n openshift jboss-webserver30-tomcat7-openshift
Name:			jboss-webserver30-tomcat7-openshift
Namespace:		openshift
Created:		30 minutes ago
Labels:			<none>
Annotations:		openshift.io/display-name=Red Hat JBoss Web Server 3.0 Apache Tomcat 7
			openshift.io/image.dockerRepositoryCheck=2019-04-11T07:39:44Z
			openshift.io/provider-display-name=Red Hat, Inc.
			version=1.4.14
Docker Pull Spec:	docker-registry.default.svc:5000/openshift/jboss-webserver30-tomcat7-openshift
Image Lookup:		local=false
Unique Images:		0
Tags:			3

1.3
  tagged from vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift:1.3
    prefer registry pullthrough when referencing this tag

  JBoss Web Server 3.0 Apache Tomcat 7 S2I images.
  Tags: builder, tomcat, tomcat7, java, jboss, hidden
  Supports: tomcat7:3.0, tomcat:7, java:8
  Example Repo: https://github.com/jboss-openshift/openshift-quickstarts.git

  ! error: Import failed (InternalError): Internal error occurred: Get https://vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/v2/: x509: certificate signed by unknown authority
      30 minutes ago

1.2
  tagged from vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift:1.2
    prefer registry pullthrough when referencing this tag

  JBoss Web Server 3.0 Apache Tomcat 7 S2I images.
  Tags: builder, tomcat, tomcat7, java, jboss, hidden
  Supports: tomcat7:3.0, tomcat:7, java:8
  Example Repo: https://github.com/jboss-openshift/openshift-quickstarts.git

  ! error: Import failed (InternalError): Internal error occurred: Get https://vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/v2/: x509: certificate signed by unknown authority
      30 minutes ago

1.1
  tagged from vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift:1.1
    prefer registry pullthrough when referencing this tag

  JBoss Web Server 3.0 Apache Tomcat 7 S2I images.
  Tags: builder, tomcat, tomcat7, java, jboss, hidden
  Supports: tomcat7:3.0, tomcat:7, java:8
  Example Repo: https://github.com/jboss-openshift/openshift-quickstarts.git

  ! error: Import failed (InternalError): Internal error occurred: Get https://vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/v2/: x509: certificate signed by unknown authority
      30 minutes ago


They are pointed to correct url.

Follow https://access.redhat.com/solutions/4026711 to import image stream image from satellite.
# oc import-image ruby -n openshift
# oc import-image jboss-webserver30-tomcat7-openshift:1.1 -n openshift
# oc import-image jboss-webserver30-tomcat7-openshift:1.2 -n openshift
# oc import-image jboss-webserver30-tomcat7-openshift:1.3 -n openshift

All the images are imported successfully.
# oc describe is -n openshift jboss-webserver30-tomcat7-openshift
Name:			jboss-webserver30-tomcat7-openshift
Namespace:		openshift
Created:		About an hour ago
Labels:			<none>
Annotations:		openshift.io/display-name=Red Hat JBoss Web Server 3.0 Apache Tomcat 7
			openshift.io/image.dockerRepositoryCheck=2019-04-11T08:59:53Z
			openshift.io/provider-display-name=Red Hat, Inc.
			version=1.4.14
Docker Pull Spec:	docker-registry.default.svc:5000/openshift/jboss-webserver30-tomcat7-openshift
Image Lookup:		local=false
Unique Images:		3
Tags:			3

1.3
  tagged from vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift:1.3
    prefer registry pullthrough when referencing this tag

  JBoss Web Server 3.0 Apache Tomcat 7 S2I images.
  Tags: builder, tomcat, tomcat7, java, jboss, hidden
  Supports: tomcat7:3.0, tomcat:7, java:8
  Example Repo: https://github.com/jboss-openshift/openshift-quickstarts.git

  * vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift@sha256:f23030e400e37ef8ba200750935f8b7a561588ced6cce74b0f03f3ee2b39f741
      38 seconds ago

1.2
  tagged from vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift:1.2
    prefer registry pullthrough when referencing this tag

  JBoss Web Server 3.0 Apache Tomcat 7 S2I images.
  Tags: builder, tomcat, tomcat7, java, jboss, hidden
  Supports: tomcat7:3.0, tomcat:7, java:8
  Example Repo: https://github.com/jboss-openshift/openshift-quickstarts.git

  * vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift@sha256:cee587ce09e25738c9fdd6cf47bb58f55cf6dc15b6f2d0ca9cdaceefbca7f8f7
      42 seconds ago

1.1
  tagged from vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift:1.1
    prefer registry pullthrough when referencing this tag

  JBoss Web Server 3.0 Apache Tomcat 7 S2I images.
  Tags: builder, tomcat, tomcat7, java, jboss, hidden
  Supports: tomcat7:3.0, tomcat:7, java:8
  Example Repo: https://github.com/jboss-openshift/openshift-quickstarts.git

  * vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-jboss-webserver-3_webserver30-tomcat7-openshift@sha256:b8dd113b8eb089cd1e888b704f64daac72ac58d7e5099674273e6ed3b2e5c4de
      About a minute ago


I also imported mongodb and nodejs image stream, trigger sti build, it succeed.
# oc get po -n install-test
NAME                             READY     STATUS      RESTARTS   AGE
mongodb-1-qfnn2                  1/1       Running     0          7m
nodejs-mongodb-example-1-build   0/1       Completed   0          6m
nodejs-mongodb-example-1-csv7m   1/1       Running     0          5m

# oc describe pod nodejs-mongodb-example-1-build -n install-test |grep -i Image:
    Image:         vm-10-0-77-71.hosted.upshift.rdu2.redhat.com:5000/default_organization-ocp3_11-disconnected-openshift3_ose-docker-builder:v3.11.88


Beside that, I also tested some other disconnected install not using satellite registry, also working well.
openshift_deployment_type=openshift-enterprise
oreg_url=vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000/testing/ocp3/ose-${component}:${version}
openshift_examples_modify_imagestreams=true
openshift_docker_insecure_registries=vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000

Installation log:
TASK [openshift_examples : Modify registry paths if registry_url is not registry.redhat.io] ***
Thursday 11 April 2019  19:00:59 +0800 (0:00:00.066)       0:08:55.852 ******** 

changed: [vm-10-0-76-121.hosted.upshift.rdu2.redhat.com] => {"changed": true, "cmd": "find /usr/share/openshift/examples -type f | xargs -n 1 sed -i 's|registry.redhat.io|vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000|g'", "delta": "0:00:00.617068", "end": "2019-04-11 07:00:59.866445", "rc": 0, "start": "2019-04-11 07:00:59.249377", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [openshift_examples : Modify registry paths if registry_url is not registry.redhat.io and using Satellite] ***
Thursday 11 April 2019  19:01:00 +0800 (0:00:01.024)       0:08:56.876 ******** 
skipping: [vm-10-0-76-121.hosted.upshift.rdu2.redhat.com] => {"changed": false, "skip_reason": "Conditional result was False"}

# oc import-image mongodb:3.4 -n openshift --insecure
# oc import-image nodejs:8 -n openshift --insecure

# oc describe is nodejs -n openshift
<--snip-->
8
  tagged from vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000/rhscl/nodejs-8-rhel7:latest
    will use insecure HTTPS or HTTP connections
    prefer registry pullthrough when referencing this tag

  Build and run Node.js 8 applications on RHEL 7. For more information about using this builder image, including OpenShift considerations, see https://github.com/sclorg/s2i-nodejs-container.
  Tags: builder, nodejs
  Example Repo: https://github.com/openshift/nodejs-ex.git

  * vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000/rhscl/nodejs-8-rhel7@sha256:073e5299478a900faf4d422f6216f0ebd7c83e85d9956806ddb0b24583ac055a
      20 seconds ago
<--snip-->


# oc describe is mongodb -n openshift
<--snip-->
3.4
  tagged from vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000/rhscl/mongodb-34-rhel7:latest
    will use insecure HTTPS or HTTP connections
    prefer registry pullthrough when referencing this tag

  Provides a MongoDB 3.4 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mongodb-container/tree/master/3.4/README.md.
  Tags: database, mongodb

  * vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000/rhscl/mongodb-34-rhel7@sha256:760c2e0af58762dba8392707ddff8a57b186d113021a7f5d4f2c15fdf07b69a5
      About a minute ago
<--snip-->

# oc get po -n install-test
NAME                             READY     STATUS      RESTARTS   AGE
mongodb-1-wwmqz                  1/1       Running     0          1m
nodejs-mongodb-example-1-2ljk5   1/1       Running     0          1m
nodejs-mongodb-example-1-build   0/1       Completed   0          1m

# oc describe pod nodejs-mongodb-example-1-build -n install-test |grep Image:
    Image:         vm-10-0-77-82.hosted.upshift.rdu2.redhat.com:5000/testing/ocp3/ose-docker-builder:v3.11.104
Comment 21 errata-xmlrpc 2019-06-06 02:00:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0794