Description of problem: When following the directions at https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0-beta/html/director_installation_and_usage/preparing-for-director-installation#preparing-container-images, installing the undercloud fails because Director cannot download the images from registry.redhat.io. It looks like the login credentials are not being passed to puppet at some point. Performing an podman login registry.redhat.com works with the same credentials and is able to pull the images. However, the undercloud install seems to wipe the local images and tries to repull them and fails. Version-Release number of selected component (if applicable): openstack-heat-common-13.0.1-0.20191127204014.0703ca7.el8ost.noarch openstack-tripleo-image-elements-10.6.1-0.20191022065313.7338463.el8ost.noarch openstack-tripleo-validations-11.3.1-0.20191126041901.2bba53a.el8ost.noarch openstack-tripleo-common-11.3.2-0.20191127200418.5c82293.el8ost.noarch openstack-heat-monolith-13.0.1-0.20191127204014.0703ca7.el8ost.noarch openstack-heat-agents-1.10.1-0.20191022061131.96b819c.el8ost.noarch openstack-tripleo-puppet-elements-11.2.1-0.20191108131052.2ad3189.el8ost.noarch openstack-heat-api-13.0.1-0.20191127204014.0703ca7.el8ost.noarch python-openstackclient-lang-4.0.0-0.20191025160014.aa64eb6.el8ost.noarch puppet-openstack_extras-15.4.1-0.20191014142330.8ba5522.el8ost.noarch puppet-openstacklib-15.4.1-0.20191014170135.94b2016.el8ost.noarch python3-openstacksdk-0.36.0-0.20191004153514.8b85e8c.el8ost.noarch openstack-heat-engine-13.0.1-0.20191127204014.0703ca7.el8ost.noarch openstack-tripleo-heat-templates-11.3.1-0.20191202212740.a4800ba.el8ost.noarch openstack-tripleo-common-containers-11.3.2-0.20191127200418.5c82293.el8ost.noarch python3-openstackclient-4.0.0-0.20191025160014.aa64eb6.el8ost.noarch openstack-ironic-python-agent-builder-1.1.1-0.20191203040321.a34dfda.el8ost.noarch ansible-role-openstack-operations-0.0.1-0.20191022044056.29cc537.el8ost.noarch openstack-selinux-0.8.20-0.20191202205815.09846a2.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. containers-prepare-parameter.yaml: parameter_defaults: ContainerImagePrepare: - set: ceph_alertmanager_image: alertmanager ceph_alertmanager_namespace: docker.io/prom ceph_alertmanager_tag: v0.16.2 ceph_grafana_image: grafana ceph_grafana_namespace: docker.io/grafana ceph_grafana_tag: 5.2.4 ceph_image: rhceph-4.0-rhel8 ceph_namespace: docker-registry.upshift.redhat.com/ceph ceph_node_exporter_image: node-exporter ceph_node_exporter_namespace: docker.io/prom ceph_node_exporter_tag: v0.17.0 ceph_prometheus_image: prometheus ceph_prometheus_namespace: docker.io/prom ceph_prometheus_tag: v2.7.2 ceph_tag: latest name_prefix: openstack- name_suffix: '' namespace: registry.redhat.io/rhosp-beta #namespace: registry.redhat.io/rhosp16 neutron_driver: ovn rhel_containers: false tag: latest tag_from_label: '{version}-{release}' ContainerImageRegistryCredentials: registry.redhat.io: 'username': 'password 2. openstack undercloud install Actual results: Exception: 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Arhosp16%2Fopenstack-cron%3Apull Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1262, in _standalone_deploy parsed_args) File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 798, in _deploy_tripleo_heat_templates self._prepare_container_images(env, roles_data) File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 743, in _prepare_container_images env, roles_data, dry_run=True) File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 217, in container_images_prepare_multi lock=lock File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 336, in container_images_prepare images, tag_from_label) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 815, in discover_image_tags discover_args): File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2264, in discover_tag_from_inspect image_url, username=username, password=password) File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f return self.call(f, *args, **kw) File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call do = self.iter(retry_state=retry_state) File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 331, in iter raise retry_exc.reraise() File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 167, in reraise raise self.last_attempt.result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call result = fn(*args, **kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 453, in authenticate rauth.raise_for_status() File "/usr/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Arhosp16%2Fopenstack-cron%3Apull None Install artifact is located at /home/stack/undercloud-install-20200116131725.tar.bzip2 Additional info: We have seen this exact issue it two completely different OpenStack 16 deployment (nothing shared between the two), so it does not appear to be user config related.
FYI, tried again with: openstack-tripleo-common noarch 11.3.3-0.20200107225621.47626e1.el8ost openstack-beta-for-rhel-8-x86_64-rpms 99 k python3-paunch noarch 5.3.1-0.20191214120154.b2f3c3c.el8ost openstack-beta-for-rhel-8-x86_64-rpms 58 k paunch-services noarch 5.3.1-0.20191214120154.b2f3c3c.el8ost openstack-beta-for-rhel-8-x86_64-rpms 17 k openstack-tripleo-heat-templates noarch 11.3.2-0.20200109050651.8f93d27.el8ost openstack-beta-for-rhel-8-x86_64-rpms 570 k openstack-tripleo-common-containers noarch 11.3.3-0.20200107225621.47626e1.el8ost openstack-beta-for-rhel-8-x86_64-rpms 37 k tripleo-ansible noarch 0.4.2-0.20200110023759.ee731ba.el8ost openstack-beta-for-rhel-8-x86_64-rpms 230 k puppet-tripleo noarch 11.4.1-0.20200106153547.5946c6f.el8ost openstack-beta-for-rhel-8-x86_64-rpms 278 k python3-tripleoclient noarch 12.3.1-0.20191230195937.585fb28.el8ost openstack-beta-for-rhel-8-x86_64-rpms 488 k python3-tripleo-common noarch 11.3.3-0.20200107225621.47626e1.el8ost openstack-beta-for-rhel-8-x86_64-rpms 294 k python3-novajoin noarch 1.3.0-0.20191217200124.265146e.el8ost openstack-beta-for-rhel-8-x86_64-rpms 109 k python3-tripleoclient-heat-installer noarch 12.3.1-0.20191230195937.585fb28.el8ost openstack-beta-for-rhel-8-x86_64-rpms 11 k
And it failed again with the Jan. 16 updates above.
The 401 is misleading here. The issue is that it is trying to fetch the GA containers which have not been published yet. Prior to GA, a custom file providing the beta container information must be used when deploying the undercloud. Did you specify the container_images_file = /home/stack/containers-prepare-parameter.yaml
So I attempted to reproduce this error with the beta and a beta containers-prepare-parameter.yaml specified for container_images_file in undercloud.conf. It didn't reproduce so I believe the issue is what I previously described where the file wasn't configured so when the install ran it attempt to pull the production containers which do not currently exist. I've proposed a bug upstream to address the error messaging around this in the future and we'll get that backported back into in a future 16 point release. If you can reproduce this with a container_images_file in undercloud.conf, please provide the undercloud.conf and the yaml file and we'll look into that further. I'll be re-purposing this bug track the error message improvements.
Created attachment 1652981 [details] undercloud.conf We *have* added container_images_file. It works fine if we provide a private registry.
I've figured it out. So if you do not specify a push_destination in the ContainerImagePrepare, you need to also specify ContainerImageRegistryLogin: true in order to have the login actually performed on the various systems. This is missing from the documentation. If you have push_destination set, we will login as part of our collection of containers to push to the undercloud registry when this is run on the undercloud during the undercloud/overcloud deployments.
The docs bz is 1792486
Adding "ContainerImageRegistryLogin: true" appears to have worked for us.
(In reply to Alex Schultz from comment #7) > I've figured it out. So if you do not specify a push_destination in the > ContainerImagePrepare, you need to also specify ContainerImageRegistryLogin: > true in order to have the login actually performed on the various systems. > This is missing from the documentation. If you have push_destination set, > we will login as part of our collection of containers to push to the > undercloud registry when this is run on the undercloud during the > undercloud/overcloud deployments. Hrm, can you add an example of a working containers-prepare-parameter.yaml file? I tried adding ContainerImageRegistryLogin to a few places in the file with no luck. Thanks!
parameter_defaults: ContainerImagePrepare: - set: ceph_alertmanager_image: alertmanager ceph_alertmanager_namespace: docker.io/prom ceph_alertmanager_tag: v0.16.2 ceph_grafana_image: grafana ceph_grafana_namespace: docker.io/grafana ceph_grafana_tag: 5.2.4 ceph_image: rhceph-4.0-rhel8 ceph_namespace: docker-registry.upshift.redhat.com/ceph ceph_node_exporter_image: node-exporter ceph_node_exporter_namespace: docker.io/prom ceph_node_exporter_tag: v0.17.0 ceph_prometheus_image: prometheus ceph_prometheus_namespace: docker.io/prom ceph_prometheus_tag: v2.7.2 ceph_tag: latest name_prefix: openstack- name_suffix: '' namespace: registry.redhat.io/rhosp-beta #namespace: registry.redhat.io/rhosp16 neutron_driver: ovn rhel_containers: false tag: latest tag_from_label: '{version}-{release}' ContainerImageRegistryLogin: true ContainerImageRegistryCredentials: registry.redhat.io: 'username': 'password'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148