Description of problem: When running openstack tripleo container image prepare. If the images returned have "Labels: None" You get the following traceback: Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/container_image.py", line 959, in take_action cleanup=parsed_args.cleanup, lock=lock) File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 217, in container_images_prepare_multi lock=lock File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 336, in container_images_prepare images, tag_from_label) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 815, in discover_image_tags discover_args): File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2275, in discover_tag_from_inspect i, image, tag_from_label, fallback_tag) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 768, in _discover_tag_from_inspect tag_label = tag_from_label.format(**labels) TypeError: format() argument after ** must be a mapping, not NoneType format() argument after ** must be a mapping, not NoneType Version-Release number of selected component (if applicable): [stack@osp16-director ~]$ rpm -qa | grep tripleo-common openstack-tripleo-common-containers-11.3.2-0.20191127200418.5c82293.el8ost.noarch python3-tripleo-common-11.3.2-0.20191127200418.5c82293.el8ost.noarch openstack-tripleo-common-11.3.2-0.20191127200418.5c82293.el8ost.noarch How reproducible: 100% Steps to Reproduce: 1. Create the containers-prepare-parameter.yaml file 2. Run sudo openstack tripleo container image prepare -e ~/containers-prepare-parameter.yaml --debug 3. If any image has Labels: None, we fail with the traceback from above. Actual results: Traceback Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/container_image.py", line 959, in take_action cleanup=parsed_args.cleanup, lock=lock) File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 217, in container_images_prepare_multi lock=lock File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 336, in container_images_prepare images, tag_from_label) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 815, in discover_image_tags discover_args): File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2275, in discover_tag_from_inspect i, image, tag_from_label, fallback_tag) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 768, in _discover_tag_from_inspect tag_label = tag_from_label.format(**labels) TypeError: format() argument after ** must be a mapping, not NoneType format() argument after ** must be a mapping, not NoneType Expected results: Can we add a if not labels: to handle this somehow? Additional info: The image that causes the issues appears to be related to Grafana: As of writing this, there is an image that has "Labels: None" as seen here: https://production.cloudflare.docker.com:443 "GET /registry-v2/docker/registry/v2/blobs/sha256/92/920eb69ade2a293782a87bc56f0b68aadb9fd0989b96f9bc88d88981aea380a1/data?verify=1576629399-ZyjmdRbpQsFlYQsfL8HhhEwffEA%3D HTTP/1.1" 200 4964 The i variable is: {'Name': 'docker.io/grafana/grafana', 'Tag': '5.2.4', 'Digest': 'sha256:aaf50da5faf2596bfb0caed81f08b5569110e7b5468b291fedad25d8cbc51f2b', 'RepoTags': ['2.0.1', '2.0.2', '2.1.0', '2.1.1', '2.1.3', '2.5.0', '2.6.0', '3$ 0.0-beta1', '3.0.0-beta3', '3.0.0-beta5', '3.0.0-beta7', '3.0.1', '3.0.2', '3.0.3-1463994644', '3.0.3', '3.0.4', '3.1.0-beta1', '3.1.0', '3.1.1-1470047149', '3.1.1', '4.0.0-beta1', '4.0.0-beta2', '4.0.0', '4.0.1-1480694114', '4.0.1', '4$ 0.2-1481203731', '4.0.2', '4.1.0-beta1', '4.1.0', '4.1.1-1484211277', '4.1.1', '4.1.2', '4.2.0-beta1', '4.2.0', '4.3.0-beta1', '4.3.0', '4.3.1', '4.3.2', '4.4.0', '4.4.1', '4.4.2', '4.4.3', '4.5.0-beta1', '4.5.0', '4.5.1', '4.5.2', '4.6$ 0-beta1', '4.6.0-beta2', '4.6.0-beta3', '4.6.0', '4.6.1', '4.6.2', '4.6.3', '4.6.4', '4.6.5', '5.0.0-beta1', '5.0.0-beta2', '5.0.0-beta3', '5.0.0-beta4', '5.0.0-beta5', '5.0.0', '5.0.1', '5.0.2', '5.0.3', '5.0.4', '5.1.0-beta1', '5.1.0'$ '5.1.1', '5.1.2', '5.1.3', '5.1.4', '5.1.5', '5.2.0-beta1', '5.2.0-beta2', '5.2.0-beta3', '5.2.0', '5.2.1', '5.2.2', '5.2.3', '5.2.4', '5.3.0-beta1', '5.3.0-beta2', '5.3.0-beta3', '5.3.0', '5.3.1', '5.3.2', '5.3.3', '5.3.4', '5.4.0-bet$ 1', '5.4.0', '5.4.1', '5.4.2', '5.4.3', '5.4.4', '5.4.5', '6.0.0-beta1', '6.0.0-beta2', '6.0.0-beta3', '6.0.0', '6.0.1', '6.0.2', '6.1.0-beta1', '6.1.0', '6.1.1', '6.1.2', '6.1.3', '6.1.4', '6.1.6', '6.2.0-beta1', '6.2.0-beta2', '6.2.0', '6.2.1', '6.2.2', '6.2.4', '6.2.5', '6.3.0-beta1', '6.3.0-beta2', '6.3.0-beta3', '6.3.0-beta4', '6.3.0', '6.3.1', '6.3.2', '6.3.3', '6.3.4', '6.3.5', '6.3.6', '6.3.7', '6.4.0-beta1', '6.4.0-beta2', '6.4.0', '6.4.1', '6.4.2', '6.4.3', '6 .4.4', '6.4.5', '6.5.0-beta1-ubuntu', '6.5.0-beta1', '6.5.0-ubuntu', '6.5.0', '6.5.1-ubuntu', '6.5.1', '6.5.2-ubuntu', '6.5.2', 'dev-musl', 'develop', 'latest-ubuntu', 'latest', 'master-ubuntu', 'master'], 'Created': '2018-09-07T14:59:30 .427058458Z', 'DockerVersion': '18.06.1-ce', 'Labels': None, 'Architecture': 'amd64', 'Os': 'linux', 'Layers': ['sha256:f2aa67a397c49232112953088506d02074a1fe577f65dc2052f158a3e5da52e8', 'sha256:e93fe8a31a9eb6193efbea8e54ef29b6bc45f59ced f6493cf116e68434046816', 'sha256:fa279080e10e2299d07f2d4599114b0fad5aae14c9de4e1847e3afdcc62344a1']}
I just changed it to be if labels and "{" in tag_from_label: which seems to have got me past the failure: /usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py: [...] 766 if labels and "{" in tag_from_label: 767 try: 768 tag_label = tag_from_label.format(**labels) 769 except ValueError as e: 770 raise ImageUploaderException(e) 771 except KeyError as e: 772 if fallback_tag: 773 tag_label = fallback_tag 774 else: 775 raise ImageUploaderException( 776 'Image %s %s. Available labels: %s' % 777 (image, e, label_keys) 778 ) [...] But we'll see if the Undercloud install works now. Feels like it should work.
My undercloud install worked after making this change. There was a little more hackery involved that is probably unrelated to this BZ, but the undercloud is now working. So I think we can review the change made here and if anyone has any other suggestions for a better idea?
Please share your containers-prepare-parameter.yaml so we can reproduce.
# Generated with the following on 2019-12-18T08:46:52.218877 # # openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml # parameter_defaults: ContainerImagePrepare: - push_destination: true set: ceph_alertmanager_image: alertmanager ceph_alertmanager_namespace: docker.io/prom ceph_alertmanager_tag: v0.16.2 ceph_grafana_image: grafana ceph_grafana_namespace: docker.io/grafana ceph_grafana_tag: 5.2.4 ceph_image: rhceph-4-rhel8 ceph_namespace: registry.redhat.io/rhceph-beta ceph_node_exporter_image: node-exporter ceph_node_exporter_namespace: docker.io/prom ceph_node_exporter_tag: v0.17.0 ceph_prometheus_image: prometheus ceph_prometheus_namespace: docker.io/prom ceph_prometheus_tag: v2.7.2 ceph_tag: latest name_prefix: openstack- name_suffix: '' namespace: registry.redhat.io/rhosp-beta neutron_driver: ovn rhel_containers: false tag: 16.0 tag_from_label: '{version}-{release}' ContainerImageRegistryCredentials: registry.redhat.io: It's basically default. But I had to change a few things, like for example by default the ceph_namespace points to upshift. :) [stack@osp16-director ~]$ diff -u default_containers-prepare-parameter.yaml containers-prepare-parameter.yaml --- default_containers-prepare-parameter.yaml 2019-12-18 11:55:25.161365908 +1000 +++ containers-prepare-parameter.yaml 2019-12-18 09:34:45.979000000 +1000 @@ -1,6 +1,6 @@ -# Generated with the following on 2019-12-18T11:55:25.155356 +# Generated with the following on 2019-12-18T08:46:52.218877 # -# openstack tripleo container image prepare default --local-push-destination --output-env-file default_containers-prepare-parameter.yaml +# openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml # parameter_defaults: @@ -13,8 +13,8 @@ ceph_grafana_image: grafana ceph_grafana_namespace: docker.io/grafana ceph_grafana_tag: 5.2.4 - ceph_image: rhceph-4.0-rhel8 - ceph_namespace: docker-registry.upshift.redhat.com/ceph + ceph_image: rhceph-4-rhel8 + ceph_namespace: registry.redhat.io/rhceph-beta ceph_node_exporter_image: node-exporter ceph_node_exporter_namespace: docker.io/prom ceph_node_exporter_tag: v0.17.0 @@ -24,8 +24,11 @@ ceph_tag: latest name_prefix: openstack- name_suffix: '' - namespace: registry.redhat.io/rhosp16 + namespace: registry.redhat.io/rhosp-beta neutron_driver: ovn rhel_containers: false tag: 16.0 tag_from_label: '{version}-{release}' + ContainerImageRegistryCredentials: + registry.redhat.io:
[stack@undercloud-0-rhosp16 ~]$ cat containers-prepare-parameter.yaml # Generated with the following on 2019-12-24T19:22:27.224663 # # openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml # parameter_defaults: ContainerImagePrepare: - push_destination: true set: ceph_image: rhceph-4-rhel8 ceph_namespace: registry.redhat.io/rhceph-beta ceph_tag: 4-8 name_prefix: openstack- name_suffix: '' namespace: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp16 neutron_driver: ovn rhel_containers: false tag: 16.0 tag_from_label: '{version}-{release}' ContainerImageRegistryCredentials: registry.redhat.io: username: "private"
I am still having traceback when running the job: https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-df-deployment-16-virthost-3cont_2comp_3ceph-ceph-ipv4-geneve-satellite-local-registry/54/artifact/ "Traceback (most recent call last):", " File \"/usr/bin/tripleo-container-image-prepare\", line 138, in <module>", " lock=lock)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py\", line 217, in container_images_prepare_multi", " lock=lock", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py\", line 336, in container_images_prepare", " images, tag_from_label)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 815, in discover_image_tags", " discover_args):", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 586, in result_iterator", " yield fs.pop().result()", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 432, in result", " return self.__get_result()", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result", " raise self._exception", " File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run", " result = self.fn(*self.args, **self.kwargs)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 2275, in discover_tag_from_inspect", " i, image, tag_from_label, fallback_tag)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 768, in _discover_tag_from_inspect", " tag_label = tag_from_label.format(**labels)", "TypeError: format() argument after ** must be a mapping, not NoneType"], "stdout": "", "stdout_lines": []} BTW, before getting this error I had a lot of errors like below one: tripleo_common.image.exception.ImageUploaderException: Image titan98.lab.eng.tlv2.redhat.com/default_organization-osp16_containers-multipathd has no tag 16.0-69.\nAvailable tags: 16.0, 16.0-1, 16.0-10, 16.0-11, 16.0-12, 16.0-13, 16.0-14, 16.0-15, 16.0-16, 16.0-17, 16.0-18, 16.0-19..... each time for different image and/or tag,
Verified on openstack-tripleo-common-11.3.2-0.20200103195912.e7e9af1
Verified by running job: https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-df-deployment-16-virthost-3cont_2comp_3ceph-ceph-ipv4-geneve-satellite-local-registry/95/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283