Bug 1784665
| Summary: | Images with Labels: None return traceback during container image prepare | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Brendan Shephard <bshephar> |
| Component: | openstack-tripleo-common | Assignee: | Emilien Macchi <emacchi> |
| Status: | CLOSED ERRATA | QA Contact: | Jad Haj Yahya <jhajyahy> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 16.0 (Train) | CC: | dhill, emacchi, jhajyahy, mburns, openstack, ramishra, Sam.Wan, sclewis, scohen, shrjoshi, slinaber |
| Target Milestone: | ga | Keywords: | Triaged |
| Target Release: | 16.0 (Train on RHEL 8.1) | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-common-11.3.3-0.20200116195535.b1b4b06.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-02-06 14:44:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I just changed it to be if labels and "{" in tag_from_label: which seems to have got me past the failure:
/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py:
[...]
766 if labels and "{" in tag_from_label:
767 try:
768 tag_label = tag_from_label.format(**labels)
769 except ValueError as e:
770 raise ImageUploaderException(e)
771 except KeyError as e:
772 if fallback_tag:
773 tag_label = fallback_tag
774 else:
775 raise ImageUploaderException(
776 'Image %s %s. Available labels: %s' %
777 (image, e, label_keys)
778 )
[...]
But we'll see if the Undercloud install works now. Feels like it should work.
My undercloud install worked after making this change. There was a little more hackery involved that is probably unrelated to this BZ, but the undercloud is now working. So I think we can review the change made here and if anyone has any other suggestions for a better idea? Please share your containers-prepare-parameter.yaml so we can reproduce. # Generated with the following on 2019-12-18T08:46:52.218877
#
# openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml
#
parameter_defaults:
ContainerImagePrepare:
- push_destination: true
set:
ceph_alertmanager_image: alertmanager
ceph_alertmanager_namespace: docker.io/prom
ceph_alertmanager_tag: v0.16.2
ceph_grafana_image: grafana
ceph_grafana_namespace: docker.io/grafana
ceph_grafana_tag: 5.2.4
ceph_image: rhceph-4-rhel8
ceph_namespace: registry.redhat.io/rhceph-beta
ceph_node_exporter_image: node-exporter
ceph_node_exporter_namespace: docker.io/prom
ceph_node_exporter_tag: v0.17.0
ceph_prometheus_image: prometheus
ceph_prometheus_namespace: docker.io/prom
ceph_prometheus_tag: v2.7.2
ceph_tag: latest
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-beta
neutron_driver: ovn
rhel_containers: false
tag: 16.0
tag_from_label: '{version}-{release}'
ContainerImageRegistryCredentials:
registry.redhat.io:
It's basically default. But I had to change a few things, like for example by default the ceph_namespace points to upshift. :)
[stack@osp16-director ~]$ diff -u default_containers-prepare-parameter.yaml containers-prepare-parameter.yaml
--- default_containers-prepare-parameter.yaml 2019-12-18 11:55:25.161365908 +1000
+++ containers-prepare-parameter.yaml 2019-12-18 09:34:45.979000000 +1000
@@ -1,6 +1,6 @@
-# Generated with the following on 2019-12-18T11:55:25.155356
+# Generated with the following on 2019-12-18T08:46:52.218877
#
-# openstack tripleo container image prepare default --local-push-destination --output-env-file default_containers-prepare-parameter.yaml
+# openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml
#
parameter_defaults:
@@ -13,8 +13,8 @@
ceph_grafana_image: grafana
ceph_grafana_namespace: docker.io/grafana
ceph_grafana_tag: 5.2.4
- ceph_image: rhceph-4.0-rhel8
- ceph_namespace: docker-registry.upshift.redhat.com/ceph
+ ceph_image: rhceph-4-rhel8
+ ceph_namespace: registry.redhat.io/rhceph-beta
ceph_node_exporter_image: node-exporter
ceph_node_exporter_namespace: docker.io/prom
ceph_node_exporter_tag: v0.17.0
@@ -24,8 +24,11 @@
ceph_tag: latest
name_prefix: openstack-
name_suffix: ''
- namespace: registry.redhat.io/rhosp16
+ namespace: registry.redhat.io/rhosp-beta
neutron_driver: ovn
rhel_containers: false
tag: 16.0
tag_from_label: '{version}-{release}'
+ ContainerImageRegistryCredentials:
+ registry.redhat.io:
[stack@undercloud-0-rhosp16 ~]$ cat containers-prepare-parameter.yaml
# Generated with the following on 2019-12-24T19:22:27.224663
#
# openstack tripleo container image prepare default --local-push-destination --output-env-file /home/stack/containers-prepare-parameter.yaml
#
parameter_defaults:
ContainerImagePrepare:
- push_destination: true
set:
ceph_image: rhceph-4-rhel8
ceph_namespace: registry.redhat.io/rhceph-beta
ceph_tag: 4-8
name_prefix: openstack-
name_suffix: ''
namespace: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp16
neutron_driver: ovn
rhel_containers: false
tag: 16.0
tag_from_label: '{version}-{release}'
ContainerImageRegistryCredentials:
registry.redhat.io:
username: "private"
I am still having traceback when running the job: https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-df-deployment-16-virthost-3cont_2comp_3ceph-ceph-ipv4-geneve-satellite-local-registry/54/artifact/ "Traceback (most recent call last):", " File \"/usr/bin/tripleo-container-image-prepare\", line 138, in <module>", " lock=lock)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py\", line 217, in container_images_prepare_multi", " lock=lock", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py\", line 336, in container_images_prepare", " images, tag_from_label)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 815, in discover_image_tags", " discover_args):", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 586, in result_iterator", " yield fs.pop().result()", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 432, in result", " return self.__get_result()", " File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result", " raise self._exception", " File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run", " result = self.fn(*self.args, **self.kwargs)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 2275, in discover_tag_from_inspect", " i, image, tag_from_label, fallback_tag)", " File \"/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py\", line 768, in _discover_tag_from_inspect", " tag_label = tag_from_label.format(**labels)", "TypeError: format() argument after ** must be a mapping, not NoneType"], "stdout": "", "stdout_lines": []} BTW, before getting this error I had a lot of errors like below one: tripleo_common.image.exception.ImageUploaderException: Image titan98.lab.eng.tlv2.redhat.com/default_organization-osp16_containers-multipathd has no tag 16.0-69.\nAvailable tags: 16.0, 16.0-1, 16.0-10, 16.0-11, 16.0-12, 16.0-13, 16.0-14, 16.0-15, 16.0-16, 16.0-17, 16.0-18, 16.0-19..... each time for different image and/or tag, Verified on openstack-tripleo-common-11.3.2-0.20200103195912.e7e9af1 Verified by running job: https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-df-deployment-16-virthost-3cont_2comp_3ceph-ceph-ipv4-geneve-satellite-local-registry/95/ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283 |
Description of problem: When running openstack tripleo container image prepare. If the images returned have "Labels: None" You get the following traceback: Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/container_image.py", line 959, in take_action cleanup=parsed_args.cleanup, lock=lock) File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 217, in container_images_prepare_multi lock=lock File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 336, in container_images_prepare images, tag_from_label) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 815, in discover_image_tags discover_args): File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2275, in discover_tag_from_inspect i, image, tag_from_label, fallback_tag) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 768, in _discover_tag_from_inspect tag_label = tag_from_label.format(**labels) TypeError: format() argument after ** must be a mapping, not NoneType format() argument after ** must be a mapping, not NoneType Version-Release number of selected component (if applicable): [stack@osp16-director ~]$ rpm -qa | grep tripleo-common openstack-tripleo-common-containers-11.3.2-0.20191127200418.5c82293.el8ost.noarch python3-tripleo-common-11.3.2-0.20191127200418.5c82293.el8ost.noarch openstack-tripleo-common-11.3.2-0.20191127200418.5c82293.el8ost.noarch How reproducible: 100% Steps to Reproduce: 1. Create the containers-prepare-parameter.yaml file 2. Run sudo openstack tripleo container image prepare -e ~/containers-prepare-parameter.yaml --debug 3. If any image has Labels: None, we fail with the traceback from above. Actual results: Traceback Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/container_image.py", line 959, in take_action cleanup=parsed_args.cleanup, lock=lock) File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 217, in container_images_prepare_multi lock=lock File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 336, in container_images_prepare images, tag_from_label) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 815, in discover_image_tags discover_args): File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2275, in discover_tag_from_inspect i, image, tag_from_label, fallback_tag) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 768, in _discover_tag_from_inspect tag_label = tag_from_label.format(**labels) TypeError: format() argument after ** must be a mapping, not NoneType format() argument after ** must be a mapping, not NoneType Expected results: Can we add a if not labels: to handle this somehow? Additional info: The image that causes the issues appears to be related to Grafana: As of writing this, there is an image that has "Labels: None" as seen here: https://production.cloudflare.docker.com:443 "GET /registry-v2/docker/registry/v2/blobs/sha256/92/920eb69ade2a293782a87bc56f0b68aadb9fd0989b96f9bc88d88981aea380a1/data?verify=1576629399-ZyjmdRbpQsFlYQsfL8HhhEwffEA%3D HTTP/1.1" 200 4964 The i variable is: {'Name': 'docker.io/grafana/grafana', 'Tag': '5.2.4', 'Digest': 'sha256:aaf50da5faf2596bfb0caed81f08b5569110e7b5468b291fedad25d8cbc51f2b', 'RepoTags': ['2.0.1', '2.0.2', '2.1.0', '2.1.1', '2.1.3', '2.5.0', '2.6.0', '3$ 0.0-beta1', '3.0.0-beta3', '3.0.0-beta5', '3.0.0-beta7', '3.0.1', '3.0.2', '3.0.3-1463994644', '3.0.3', '3.0.4', '3.1.0-beta1', '3.1.0', '3.1.1-1470047149', '3.1.1', '4.0.0-beta1', '4.0.0-beta2', '4.0.0', '4.0.1-1480694114', '4.0.1', '4$ 0.2-1481203731', '4.0.2', '4.1.0-beta1', '4.1.0', '4.1.1-1484211277', '4.1.1', '4.1.2', '4.2.0-beta1', '4.2.0', '4.3.0-beta1', '4.3.0', '4.3.1', '4.3.2', '4.4.0', '4.4.1', '4.4.2', '4.4.3', '4.5.0-beta1', '4.5.0', '4.5.1', '4.5.2', '4.6$ 0-beta1', '4.6.0-beta2', '4.6.0-beta3', '4.6.0', '4.6.1', '4.6.2', '4.6.3', '4.6.4', '4.6.5', '5.0.0-beta1', '5.0.0-beta2', '5.0.0-beta3', '5.0.0-beta4', '5.0.0-beta5', '5.0.0', '5.0.1', '5.0.2', '5.0.3', '5.0.4', '5.1.0-beta1', '5.1.0'$ '5.1.1', '5.1.2', '5.1.3', '5.1.4', '5.1.5', '5.2.0-beta1', '5.2.0-beta2', '5.2.0-beta3', '5.2.0', '5.2.1', '5.2.2', '5.2.3', '5.2.4', '5.3.0-beta1', '5.3.0-beta2', '5.3.0-beta3', '5.3.0', '5.3.1', '5.3.2', '5.3.3', '5.3.4', '5.4.0-bet$ 1', '5.4.0', '5.4.1', '5.4.2', '5.4.3', '5.4.4', '5.4.5', '6.0.0-beta1', '6.0.0-beta2', '6.0.0-beta3', '6.0.0', '6.0.1', '6.0.2', '6.1.0-beta1', '6.1.0', '6.1.1', '6.1.2', '6.1.3', '6.1.4', '6.1.6', '6.2.0-beta1', '6.2.0-beta2', '6.2.0', '6.2.1', '6.2.2', '6.2.4', '6.2.5', '6.3.0-beta1', '6.3.0-beta2', '6.3.0-beta3', '6.3.0-beta4', '6.3.0', '6.3.1', '6.3.2', '6.3.3', '6.3.4', '6.3.5', '6.3.6', '6.3.7', '6.4.0-beta1', '6.4.0-beta2', '6.4.0', '6.4.1', '6.4.2', '6.4.3', '6 .4.4', '6.4.5', '6.5.0-beta1-ubuntu', '6.5.0-beta1', '6.5.0-ubuntu', '6.5.0', '6.5.1-ubuntu', '6.5.1', '6.5.2-ubuntu', '6.5.2', 'dev-musl', 'develop', 'latest-ubuntu', 'latest', 'master-ubuntu', 'master'], 'Created': '2018-09-07T14:59:30 .427058458Z', 'DockerVersion': '18.06.1-ce', 'Labels': None, 'Architecture': 'amd64', 'Os': 'linux', 'Layers': ['sha256:f2aa67a397c49232112953088506d02074a1fe577f65dc2052f158a3e5da52e8', 'sha256:e93fe8a31a9eb6193efbea8e54ef29b6bc45f59ced f6493cf116e68434046816', 'sha256:fa279080e10e2299d07f2d4599114b0fad5aae14c9de4e1847e3afdcc62344a1']}