Bug 1791784

Summary: OSP 16 beta fails when performing an authenticated pull from registry.redhat.io
Product: Red Hat OpenStack Reporter: Brian J. Atkisson <batkisso>
Component: openstack-tripleo-commonAssignee: Alex Schultz <aschultz>
Status: CLOSED ERRATA QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 16.0 (Train)CC: aschultz, augol, emacchi, hbrock, jschluet, jslagle, juvillar, mburns, mchappel, slinaber
Target Milestone: betaKeywords: Reopened, Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-11.3.3-0.20200302223724.1e95c34.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-29 07:50:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1792486    
Bug Blocks:    
Attachments:
Description Flags
undercloud.conf none

Description Brian J. Atkisson 2020-01-16 13:20:52 UTC
Description of problem:

When following the directions at https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0-beta/html/director_installation_and_usage/preparing-for-director-installation#preparing-container-images, installing the undercloud fails because Director cannot download the images from registry.redhat.io. It looks like the login credentials are not being passed to puppet at some point. Performing an podman login registry.redhat.com works with the same credentials and is able to pull the images. However, the undercloud install seems to wipe the local images and tries to repull them and fails.



Version-Release number of selected component (if applicable):
openstack-heat-common-13.0.1-0.20191127204014.0703ca7.el8ost.noarch
openstack-tripleo-image-elements-10.6.1-0.20191022065313.7338463.el8ost.noarch
openstack-tripleo-validations-11.3.1-0.20191126041901.2bba53a.el8ost.noarch
openstack-tripleo-common-11.3.2-0.20191127200418.5c82293.el8ost.noarch
openstack-heat-monolith-13.0.1-0.20191127204014.0703ca7.el8ost.noarch
openstack-heat-agents-1.10.1-0.20191022061131.96b819c.el8ost.noarch
openstack-tripleo-puppet-elements-11.2.1-0.20191108131052.2ad3189.el8ost.noarch
openstack-heat-api-13.0.1-0.20191127204014.0703ca7.el8ost.noarch
python-openstackclient-lang-4.0.0-0.20191025160014.aa64eb6.el8ost.noarch
puppet-openstack_extras-15.4.1-0.20191014142330.8ba5522.el8ost.noarch
puppet-openstacklib-15.4.1-0.20191014170135.94b2016.el8ost.noarch
python3-openstacksdk-0.36.0-0.20191004153514.8b85e8c.el8ost.noarch
openstack-heat-engine-13.0.1-0.20191127204014.0703ca7.el8ost.noarch
openstack-tripleo-heat-templates-11.3.1-0.20191202212740.a4800ba.el8ost.noarch
openstack-tripleo-common-containers-11.3.2-0.20191127200418.5c82293.el8ost.noarch
python3-openstackclient-4.0.0-0.20191025160014.aa64eb6.el8ost.noarch
openstack-ironic-python-agent-builder-1.1.1-0.20191203040321.a34dfda.el8ost.noarch
ansible-role-openstack-operations-0.0.1-0.20191022044056.29cc537.el8ost.noarch
openstack-selinux-0.8.20-0.20191202205815.09846a2.el8ost.noarch

How reproducible:
Always

Steps to Reproduce:
1.

containers-prepare-parameter.yaml:
parameter_defaults:
  ContainerImagePrepare:
  - set:
      ceph_alertmanager_image: alertmanager
      ceph_alertmanager_namespace: docker.io/prom
      ceph_alertmanager_tag: v0.16.2
      ceph_grafana_image: grafana
      ceph_grafana_namespace: docker.io/grafana
      ceph_grafana_tag: 5.2.4
      ceph_image: rhceph-4.0-rhel8
      ceph_namespace: docker-registry.upshift.redhat.com/ceph
      ceph_node_exporter_image: node-exporter
      ceph_node_exporter_namespace: docker.io/prom
      ceph_node_exporter_tag: v0.17.0
      ceph_prometheus_image: prometheus
      ceph_prometheus_namespace: docker.io/prom
      ceph_prometheus_tag: v2.7.2
      ceph_tag: latest
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-beta
      #namespace: registry.redhat.io/rhosp16
      neutron_driver: ovn
      rhel_containers: false
      tag: latest
    tag_from_label: '{version}-{release}'
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      'username': 'password


2. openstack undercloud install

Actual results:

Exception: 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Arhosp16%2Fopenstack-cron%3Apull
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1262, in _standalone_deploy
    parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 798, in _deploy_tripleo_heat_templates
    self._prepare_container_images(env, roles_data)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 743, in _prepare_container_images
    env, roles_data, dry_run=True)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 217, in container_images_prepare_multi
    lock=lock
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 336, in container_images_prepare
    images, tag_from_label)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 815, in discover_image_tags
    discover_args):
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2264, in discover_tag_from_inspect
    image_url, username=username, password=password)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 331, in iter
    raise retry_exc.reraise()
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 167, in reraise
    raise self.last_attempt.result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 453, in authenticate
    rauth.raise_for_status()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Arhosp16%2Fopenstack-cron%3Apull
None
Install artifact is located at /home/stack/undercloud-install-20200116131725.tar.bzip2



Additional info:

We have seen this exact issue it two completely different OpenStack 16 deployment (nothing shared between the two), so it does not appear to be user config related.

Comment 1 Brian J. Atkisson 2020-01-16 15:27:27 UTC
FYI, tried again with:
 openstack-tripleo-common                        noarch            11.3.3-0.20200107225621.47626e1.el8ost               openstack-beta-for-rhel-8-x86_64-rpms             99 k
 python3-paunch                                  noarch            5.3.1-0.20191214120154.b2f3c3c.el8ost                openstack-beta-for-rhel-8-x86_64-rpms             58 k
 paunch-services                                 noarch            5.3.1-0.20191214120154.b2f3c3c.el8ost                openstack-beta-for-rhel-8-x86_64-rpms             17 k
 openstack-tripleo-heat-templates                noarch            11.3.2-0.20200109050651.8f93d27.el8ost               openstack-beta-for-rhel-8-x86_64-rpms            570 k
 openstack-tripleo-common-containers             noarch            11.3.3-0.20200107225621.47626e1.el8ost               openstack-beta-for-rhel-8-x86_64-rpms             37 k
 tripleo-ansible                                 noarch            0.4.2-0.20200110023759.ee731ba.el8ost                openstack-beta-for-rhel-8-x86_64-rpms            230 k
 puppet-tripleo                                  noarch            11.4.1-0.20200106153547.5946c6f.el8ost               openstack-beta-for-rhel-8-x86_64-rpms            278 k
 python3-tripleoclient                           noarch            12.3.1-0.20191230195937.585fb28.el8ost               openstack-beta-for-rhel-8-x86_64-rpms            488 k
 python3-tripleo-common                          noarch            11.3.3-0.20200107225621.47626e1.el8ost               openstack-beta-for-rhel-8-x86_64-rpms            294 k
 python3-novajoin                                noarch            1.3.0-0.20191217200124.265146e.el8ost                openstack-beta-for-rhel-8-x86_64-rpms            109 k
 python3-tripleoclient-heat-installer            noarch            12.3.1-0.20191230195937.585fb28.el8ost               openstack-beta-for-rhel-8-x86_64-rpms             11 k

Comment 2 Brian J. Atkisson 2020-01-16 15:28:06 UTC
And it failed again with the Jan. 16 updates above.

Comment 3 Alex Schultz 2020-01-16 19:58:24 UTC
The 401 is misleading here. The issue is that it is trying to fetch the GA containers which have not been published yet. Prior to GA, a custom file providing the beta container information must be used when deploying the undercloud.  Did you specify the container_images_file = /home/stack/containers-prepare-parameter.yaml

Comment 4 Alex Schultz 2020-01-16 22:00:33 UTC
So I attempted to reproduce this error with the beta and a beta containers-prepare-parameter.yaml specified for container_images_file in undercloud.conf. It didn't reproduce so I believe the issue is what I previously described where the file wasn't configured so when the install ran it attempt to pull the production containers which do not currently exist. I've proposed a bug upstream to address the error messaging around this in the future and we'll get that backported back into in a future 16 point release.  If you can reproduce this with a container_images_file in undercloud.conf, please provide the undercloud.conf and the yaml file and we'll look into that further.  I'll be re-purposing this bug track the error message improvements.

Comment 5 Mark Chappell 2020-01-17 08:54:11 UTC
Created attachment 1652981 [details]
undercloud.conf

We *have* added container_images_file.  It works fine if we provide a private registry.

Comment 7 Alex Schultz 2020-01-17 17:17:07 UTC
I've figured it out. So if you do not specify a push_destination in the ContainerImagePrepare, you need to also specify ContainerImageRegistryLogin: true in order to have the login actually performed on the various systems.  This is missing from the documentation.  If you have push_destination set, we will login as part of our collection of containers to push to the undercloud registry when this is run on the undercloud during the undercloud/overcloud deployments.

Comment 8 Alex Schultz 2020-01-17 17:39:54 UTC
The docs bz is 1792486

Comment 9 Mark Chappell 2020-01-20 08:48:57 UTC
Adding "ContainerImageRegistryLogin: true" appears to have worked for us.

Comment 10 Brian J. Atkisson 2020-01-21 04:15:34 UTC
(In reply to Alex Schultz from comment #7)
> I've figured it out. So if you do not specify a push_destination in the
> ContainerImagePrepare, you need to also specify ContainerImageRegistryLogin:
> true in order to have the login actually performed on the various systems. 
> This is missing from the documentation.  If you have push_destination set,
> we will login as part of our collection of containers to push to the
> undercloud registry when this is run on the undercloud during the
> undercloud/overcloud deployments.


Hrm, can you add an example of a working containers-prepare-parameter.yaml file? I tried adding ContainerImageRegistryLogin to a few places in the file with no luck. Thanks!

Comment 11 Alex Schultz 2020-01-21 15:25:34 UTC
parameter_defaults:
  ContainerImagePrepare:
  - set:
      ceph_alertmanager_image: alertmanager
      ceph_alertmanager_namespace: docker.io/prom
      ceph_alertmanager_tag: v0.16.2
      ceph_grafana_image: grafana
      ceph_grafana_namespace: docker.io/grafana
      ceph_grafana_tag: 5.2.4
      ceph_image: rhceph-4.0-rhel8
      ceph_namespace: docker-registry.upshift.redhat.com/ceph
      ceph_node_exporter_image: node-exporter
      ceph_node_exporter_namespace: docker.io/prom
      ceph_node_exporter_tag: v0.17.0
      ceph_prometheus_image: prometheus
      ceph_prometheus_namespace: docker.io/prom
      ceph_prometheus_tag: v2.7.2
      ceph_tag: latest
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-beta
      #namespace: registry.redhat.io/rhosp16
      neutron_driver: ovn
      rhel_containers: false
      tag: latest
    tag_from_label: '{version}-{release}'
  ContainerImageRegistryLogin: true
  ContainerImageRegistryCredentials:
    registry.redhat.io:
      'username': 'password'

Comment 18 errata-xmlrpc 2020-07-29 07:50:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148