Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1632414

Summary: Default imagestreams import images from the undercloud
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Martin André <m.andre>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 14.0 (Rocky)CC: dbecker, jtrowbri, m.andre, mburns, mcornea, morazi, racedoro, sasha
Target Milestone: betaKeywords: Triaged
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060864.ffbe879.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 11:53:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2018-09-24 18:07:21 UTC
Description of problem:
Default imagestreams import images from the undercloud which fails because the undercloud registry doesn't contain such images:

After successful overcloud deployment:

[cloud-user@openshift-tester-0 ~]$ oc describe imagestreams mysql --namespace=openshift
Name:			mysql
Namespace:		openshift
Created:		47 hours ago
Labels:			<none>
Annotations:		openshift.io/display-name=MySQL
			openshift.io/image.dockerRepositoryCheck=2018-09-22T18:38:45Z
Docker Pull Spec:	docker-registry.default.svc:5000/openshift/mysql
Image Lookup:		local=false
Unique Images:		0
Tags:			4

5.7 (latest)
  tagged from 192.168.24.1:8787/rhscl/mysql-57-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a MySQL 5.7 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mysql-container/tree/master/5.7/README.md.
  Tags: mysql

  ! error: Import failed (InternalError): Internal error occurred: Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client
      47 hours ago

5.6
  tagged from 192.168.24.1:8787/rhscl/mysql-56-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a MySQL 5.6 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mysql-container/tree/master/5.6/README.md.
  Tags: hidden, mysql

  ! error: Import failed (InternalError): Internal error occurred: Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client
      47 hours ago

5.5
  tagged from 192.168.24.1:8787/openshift3/mysql-55-rhel7:latest
    prefer registry pullthrough when referencing this tag

  Provides a MySQL 5.5 database on RHEL 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mysql-container/tree/master/5.5/README.md.
  Tags: hidden, mysql

  ! error: Import failed (InternalError): Internal error occurred: Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client
      47 hours ago


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-9.0.0-0.20180906145841.66804ff.0rc1.0rc1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud:

openstack overcloud deploy \
--stack openshift \
--templates \
-r /home/stack/openshift_roles_data.yaml \
-n /home/stack/network_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/openshift.yaml \
-e /home/stack/openshift_env.yaml \
-e /home/stack/containers-default-parameters.yaml

2. Check imagestreams on master node:
oc get imagestreams --namespace=openshift
oc describe imagestreams mysql --namespace=openshift


Actual results:
Imagestreams try to import images from undercloud where they do not exist.

Expected results:
Imagestreams import images from valid source.

Additional info:

Environment files used during deployment:

(undercloud) [stack@undercloud-0 ~]$ cat /home/stack/openshift_env.yaml
resource_registry:
  OS::TripleO::Services::HAproxy: /usr/share/openstack-tripleo-heat-templates/docker/services/haproxy.yaml
  OS::TripleO::Services::Keepalived: /usr/share/openstack-tripleo-heat-templates//docker/services/keepalived.yaml
  OS::TripleO::NodeUserData: /home/stack/firstboot.yaml
  OS::TripleO::OpenShiftMaster::Net::SoftwareConfig: /home/stack/master-nic.yaml
  OS::TripleO::OpenShiftWorker::Net::SoftwareConfig: /home/stack/worker-nic.yaml
  OS::TripleO::OpenShiftInfra::Net::SoftwareConfig: /home/stack/infra-nic.yaml

parameter_defaults:
  CloudName: openshift.localdomain

  OvercloudOpenShiftMasterFlavor: master
  OpenShiftMasterHostnameFormat: '%stackname%-master-%index%'
  OvercloudOpenShiftWorkerFlavor: worker
  OpenShiftWorkerHostnameFormat: '%stackname%-worker-%index%'
  OvercloudOpenShiftInfraFlavor: infra
  OpenShiftInfraHostnameFormat: '%stackname%-infra-%index%'

  OpenShiftMasterCount: 3
  OpenShiftWorkerCount: 2
  OpenShiftInfraCount: 2

  NtpServer: ["clock.redhat.com","clock2.redhat.com"]

  ControlPlaneDefaultRoute: 192.168.24.1
  EC2MetadataIp: 192.168.24.1
  ControlPlaneSubnetCidr: 24

  DnsServers:
   - 10.0.0.1

  # NOTE(flaper87): This should be 3.10
  # eventually
  OpenShiftGlobalVariables:

    # Allow all auth
    # https://docs.openshift.com/container-platform/3.7/install_config/configuring_authentication.html#overview
    openshift_master_identity_providers:
    - name: allow_all
      login: 'true'
      challenge: true
      kind: AllowAllPasswordIdentityProvider

    # NOTE(flaper87): Needed for the gate
    openshift_disable_check: package_availability,package_version,disk_availability,docker_storage,memory_availability,docker_image_availability
    openshift_deployment_type: openshift-enterprise
(undercloud) [stack@undercloud-0 ~]$ cat /home/stack/containers-default-parameters.yaml
# Generated with the following on 2018-09-22T14:00:46.708173
#
#   openstack tripleo container image prepare -e /home/stack/containers-prepare-parameter.yaml --roles-file /home/stack/openshift_roles_data.yaml --output-env-file /home/stack/containers-default-parameters.yaml
#

parameter_defaults:
  DockerHAProxyConfigImage: 192.168.24.1:8787/rhosp14/openstack-haproxy:2018-09-20.1
  DockerHAProxyImage: 192.168.24.1:8787/rhosp14/openstack-haproxy:2018-09-20.1
  DockerInsecureRegistryAddress:
  - 192.168.24.1:8787
  DockerKeepalivedConfigImage: 192.168.24.1:8787/rhosp14/openstack-keepalived:2018-09-20.1
  DockerKeepalivedImage: 192.168.24.1:8787/rhosp14/openstack-keepalived:2018-09-20.1
  DockerOpenShiftBaseImage: 192.168.24.1:8787/openshift3/ose:v3.10
  DockerOpenShiftCockpitImage: 192.168.24.1:8787/openshift3/registry-console:v3.10
  DockerOpenShiftDeployerImage: 192.168.24.1:8787/openshift3/ose-deployer:v3.10
  DockerOpenShiftDockerRegistryImage: 192.168.24.1:8787/openshift3/ose-docker-registry:v3.10
  DockerOpenShiftEtcdImage: 192.168.24.1:8787/rhel7/etcd:latest
  DockerOpenShiftGlusterFSBlockImage: 192.168.24.1:8787/rhgs3/rhgs-gluster-block-prov-rhel7:latest
  DockerOpenShiftGlusterFSHeketiImage: 192.168.24.1:8787/rhgs3/rhgs-volmanager-rhel7:latest
  DockerOpenShiftGlusterFSImage: 192.168.24.1:8787/rhgs3/rhgs-server-rhel7:latest
  DockerOpenShiftHAProxyRouterImage: 192.168.24.1:8787/openshift3/ose-haproxy-router:v3.10
  DockerOpenShiftNodeImage: 192.168.24.1:8787/openshift3/node:v3.10
  DockerOpenShiftPodImage: 192.168.24.1:8787/openshift3/ose-pod:v3.10
  DockerOpenShiftWebConsoleImage: 192.168.24.1:8787/openshift3/ose-web-console:v3.10

Comment 1 Martin André 2018-10-04 16:57:45 UTC
I guess this has to do with different default for upstream vs downstream openshift-ansible. Here is what I get when deploying openshift using the latest upstream patches:

[heat-admin@openshift-openshiftmaster-0 ~]$ sudo oc describe imagestreams mysql --namespace=openshift
Name:                   mysql
Namespace:              openshift
Created:                38 minutes ago
Labels:                 <none>
Annotations:            openshift.io/display-name=MySQL
                        openshift.io/image.dockerRepositoryCheck=2018-10-04T16:13:58Z
Docker Pull Spec:       docker-registry.default.svc:5000/openshift/mysql
Image Lookup:           local=false
Unique Images:          3
Tags:                   4

5.7 (latest)
  tagged from docker.io/centos/mysql-57-centos7:latest
    prefer registry pullthrough when referencing this tag

  Provides a MySQL 5.7 database on CentOS 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mysql-container/tree/master/5.7/README.md.
  Tags: mysql

  * docker.io/centos/mysql-57-centos7@sha256:063906bd5d5f6dc9218b6a598160b17d051d663f7eed23b3e1fdcf225b257ae7
      38 minutes ago

5.6
  tagged from docker.io/centos/mysql-56-centos7:latest
    prefer registry pullthrough when referencing this tag

  Provides a MySQL 5.6 database on CentOS 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mysql-container/tree/master/5.6/README.md.
  Tags: hidden, mysql

  * docker.io/centos/mysql-56-centos7@sha256:ad629a897185728349b33073300bb657303a0df14067bab323d72416e31f4f0f
      38 minutes ago

5.5
  tagged from docker.io/openshift/mysql-55-centos7:latest
    prefer registry pullthrough when referencing this tag

  Provides a MySQL 5.5 database on CentOS 7. For more information about using this database image, including OpenShift considerations, see https://github.com/sclorg/mysql-container/tree/master/5.5/README.md.
  Tags: hidden, mysql

  * docker.io/openshift/mysql-55-centos7@sha256:7cd33ccc63b8005810aedee0444472cc84c3e3c38fe9cd147edb964e712a1068
      38 minutes ago


I'll try to deploy openshift using a downstream build of openshift-ansible see if it makes a difference.

Comment 4 Martin André 2018-10-09 13:36:01 UTC
It's very likely due to the openshift_examples_modify_imagestreams variable we set to true in the tripleo-heat-templates which cause openshift-ansible to rewrite the registry for the imagestream.

https://github.com/openshift/openshift-ansible/blob/95bc2d2e61f23bfc81e1160ad7fc0a8ffef6c2d9/roles/openshift_examples/tasks/main.yml#L52-L55

I'm thinking we should leave this variable unset.

Comment 5 Martin André 2018-10-10 15:10:04 UTC
So leaving the openshift_examples_modify_imagestreams variable to its default value seems to do the trick.

I've submitted https://review.openstack.org/#/c/609445/ for it.

Comment 13 errata-xmlrpc 2019-01-11 11:53:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045