Description of problem: Our deployment (composable control plane) is failing with: image: http://rhos-qe-mirror-rdu2.usersys.redhat.com:5002/v2/rh-osbs/rhosp16-openstack-sahara-api/manifests/16.2_20201008.1", "\"\"\"", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", " File \"/usr/lib/python3.6/site-packages/tripleoclient/command.py\", line 32, in run", " super(Command, self).run(parsed_args)", " File \"/usr/lib/python3.6/site-packages/osc_lib/command/command.py\", line 41, in run", " return super(Command, self).run(parsed_args)", " File \"/usr/lib/python3.6/site-packages/cliff/command.py\", line 185, in run", " return_code = self.take_action(parsed_args) or 0", " File \"/usr/lib/python3.6/site-packages/tripleoclient/v1/container_image.py\", line 1055, in take_action", " cleanup=parsed_args.cleanup, lock=lock)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py\", line 241, in container_images_prepare_multi", " uploader.upload()", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 506, in upload", " uploader.run_tasks()", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 2535, in run_tasks", " for result in p.map(upload_task, self.upload_tasks):", " File \"/usr/lib64/python3.6/concurrent/futures/process.py\", line 366, in _chain_from_iterable_of_lists", " for element in iterable:", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 586, in result_iterator", " yield fs.pop().result()", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result", " return self.__get_result()", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result", " raise self._exception", "tripleo_common.image.exception.ImageNotFoundException: Not found image: http://rhos-qe-mirror-rdu2.usersys.redhat.com:5002/v2/rh-osbs/rhosp16-openstack-sahara-api/manifests/16.2_20201008.1", "Not found image: http://rhos-qe-mirror-rdu2.usersys.redhat.com:5002/v2/rh-osbs/rhosp16-openstack-sahara-api/manifests/16.2_20201008.1"], "stdout": "", "stdout_lines": []}[0m I don't think it is a mirror problem since the image is not there both on the rdu mirror nor on the brq mirror: [stack@undercloud-0 ~]$ sudo podman pull rhos-qe-mirror-brq.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1 Trying to pull rhos-qe-mirror-brq.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1... manifest unknown: manifest unknown Error: unable to pull rhos-qe-mirror-brq.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1: Error initializing source docker://rhos-qe-mirror-brq.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1: Error reading manifest 16.2_20201008.1 in rhos-qe-mirror-brq.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api: manifest unknown: manifest unknown [stack@undercloud-0 ~]$ sudo podman pull rhos-qe-mirror-rdu2.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1 Trying to pull rhos-qe-mirror-rdu2.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1... manifest unknown: manifest unknown Error: unable to pull rhos-qe-mirror-rdu2.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1: Error initializing source docker://rhos-qe-mirror-rdu2.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api:16.2_20201008.1: Error reading manifest 16.2_2 0201008.1 in rhos-qe-mirror-rdu2.usersys.redhat.com:5002/rh-osbs/rhosp16-openstack-sahara-api: manifest unknown: manifest unknown Via https://code.engineering.redhat.com/gerrit/#/c/176688/ we dropped them downstream, yet the overcloud deployment (UC succeeds) fails because it is trying to pull them in. My first hunch is that https://code.engineering.redhat.com/gerrit/#/c/176688/ is not patching container-images/tripleo_containers.yaml.j2 What I don't know yet is why none else is hitting this. I'll keep poking around and assign it to me for the time being
Testing https://code.engineering.redhat.com/gerrit/#/c/214642/
So https://code.engineering.redhat.com/gerrit/#/c/214642/ fixes it for me. What remains unexplained is why none else is hitting this.
*** Bug 1896436 has been marked as a duplicate of this bug. ***
*** Bug 1899194 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.2 containers bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3489