Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHEA-2019:2811
Description of problem: While working on porting OSP15 to RHEL8 we hit the following issue on the undercloud with glance running in a python3 based container and talking to swift (which runs via python2 inside another container). This is the full error message: 2019-01-16 16:37:05.079 50 DEBUG glance_store._drivers.swift.store [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Adding image object '42d555ef-3757-4b6e-b91f-0969d6c8302a' to Swift add /usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py:941 2019-01-16 16:37:05.079 50 DEBUG glance_store._drivers.swift.store [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift. add /usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py:964 2019-01-16 16:37:06.560 50 ERROR glance_store._drivers.swift.store [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Error during chunked upload to backend, deleting stale chunks. 2019-01-16 16:37:06.566 50 ERROR glance.api.v2.image_data [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Failed to upload image data due to internal error 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Caught error: Unicode-objects must be encoded before hashing: TypeError: Unicode-objects must be encoded before hashing 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi Traceback (most recent call last): 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1253, in __call__ 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi request, **action_args) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1296, in dispatch 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi return method(*args, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 417, in wrapped 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 279, in upload 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self._restore(image_repo, image) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.force_reraise() 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi raise value 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 144, in upload 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 198, in set_data 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 485, in set_data 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.force_reraise() 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi raise value 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 432, in set_data 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 193, in set_data 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 318, in set_data 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 483, in set_data 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi verifier=verifier) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/backend.py", line 541, in add_to_backend_with_multihash 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/backend.py", line 518, in store_add_to_backend_with_multihash 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 274, in add_adapter 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 225, in op_checker 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1012, in add 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi written_chunks) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.force_reraise() 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi raise value 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1001, in add 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi content_length=content_length) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1878, in put_object 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi response_dict=response_dict) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1722, in _retry 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi service_token=self.service_token, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1346, in put_object 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi conn.putrequest(path, headers=headers, data=data) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 454, in putrequest 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi return self.request('PUT', full_path, data, headers, files) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 444, in request 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi files=files, **self.requests_args) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 427, in _request 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi return self.request_session.request(*arg, **kwarg) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi resp = self.send(prep, **send_kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi r = adapter.send(request, **kwargs) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 469, in send 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi for i in request.body: 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/swiftclient/utils.py", line 298, in __next__ 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi chunk = self.content.read(self.chunk_size) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1636, in read 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi self.checksum.update(result) 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi TypeError: Unicode-objects must be encoded before hashing 2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi 2019-01-16 16:37:06.672 50 INFO eventlet.wsgi.server [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] 192.168.24.2 - - [16/Jan/2019 16:37:06] "PUT /v2/images/42d555ef-3757-4b6e-b91f-0969d6c8302a/file HTTP/1.1" 500 454 2.705659 Upstream bug seems to be https://bugs.launchpad.net/glance/+bug/1772018 The following review fixes it for us: https://review.openstack.org/#/c/620234/ but there seems to be some disagreement over the correctness of the fix. The versions used where we saw this are: python3-glanceclient-2.15.0-0.20181226111056.c4c92ec.fc28.noarch python3-glance-17.1.0-0.20190109134015.fb2c2e7.fc28.noarch openstack-glance-17.1.0-0.20190109134015.fb2c2e7.fc28.noarch python3-glance-store-0.27.1-0.20190108172526.de27161.fc28.noarch