1677658 – Running glance via python3 breaks when using the swift backend

Bug 1677658 - Running glance via python3 breaks when using the swift backend

Summary: Running glance via python3 breaks when using the swift backend

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-glance-store
Sub Component:
Version:	15.0 (Stein)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	beta
Target Release:	15.0 (Stein)
Assignee:	Cyril Roelandt
QA Contact:	Mike Abrams
Docs Contact:	Tana
URL:
Whiteboard:
Duplicates (2):	1689432 1689989 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-02-15 13:50 UTC by Michele Baldessari
Modified:	2021-07-03 02:25 UTC (History)
CC List:	10 users (show)
Fixed In Version:	python-glance-store-0.28.0-0.20190301020712.5b68597.1.el8ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-09-21 11:20:21 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1772018	None	None	None	2019-03-01 08:59:15 UTC
OpenStack gerrit	620234	None	master: NEW	glance_store: encode image data if it's not bytes (Ia7653f9fcbe902abc203c10c80ab44a641a4d8f9)	2019-03-01 08:57:31 UTC
OpenStack gerrit	644839	None	None	None	2019-03-20 12:05:34 UTC
Red Hat Product Errata	RHEA-2019:2811	None	None	None	2019-09-21 11:20:38 UTC

Description Michele Baldessari 2019-02-15 13:50:35 UTC

Description of problem:
While working on porting OSP15 to RHEL8 we hit the following issue on the undercloud with glance running in a python3 based container and talking to swift (which runs via python2 inside another container). This is the full error message:
2019-01-16 16:37:05.079 50 DEBUG glance_store._drivers.swift.store [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Adding image object '42d555ef-3757-4b6e-b91f-0969d6c8302a' to Swift add /usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py:941
2019-01-16 16:37:05.079 50 DEBUG glance_store._drivers.swift.store [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift. add /usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py:964
2019-01-16 16:37:06.560 50 ERROR glance_store._drivers.swift.store [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Error during chunked upload to backend, deleting stale chunks.
2019-01-16 16:37:06.566 50 ERROR glance.api.v2.image_data [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Failed to upload image data due to internal error
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] Caught error: Unicode-objects must be encoded before hashing: TypeError: Unicode-objects must be encoded before hashing
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi Traceback (most recent call last):
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1253, in __call__
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     request, **action_args)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1296, in dispatch
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     return method(*args, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 417, in wrapped
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 279, in upload
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self._restore(image_repo, image)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.force_reraise()
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     raise value
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 144, in upload
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 198, in set_data
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.base.set_data(data, size, backend=backend)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 485, in set_data
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.force_reraise()
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     raise value
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 432, in set_data
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.repo.set_data(data, size, backend=backend)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 193, in set_data
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 318, in set_data
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.image.set_data(data, size=size, backend=backend)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/location.py", line 483, in set_data
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     verifier=verifier)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/backend.py", line 541, in add_to_backend_with_multihash
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     image_id, data, size, hashing_algo, store, context, verifier)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/backend.py", line 518, in store_add_to_backend_with_multihash
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     image_id, data, size, hashing_algo, context=context, verifier=verifier)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 274, in add_adapter
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     metadata_dict) = store_add_fun(*args, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 225, in op_checker
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1012, in add
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     written_chunks)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.force_reraise()
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     raise value
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1001, in add
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     content_length=content_length)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1878, in put_object
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     response_dict=response_dict)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1722, in _retry
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     service_token=self.service_token, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1346, in put_object
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     conn.putrequest(path, headers=headers, data=data)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 454, in putrequest
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     return self.request('PUT', full_path, data, headers, files)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 444, in request
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     files=files, **self.requests_args)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 427, in _request
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     return self.request_session.request(*arg, **kwarg)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     resp = self.send(prep, **send_kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     r = adapter.send(request, **kwargs)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 469, in send
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     for i in request.body:
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/swiftclient/utils.py", line 298, in __next__
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     chunk = self.content.read(self.chunk_size)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1636, in read
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi     self.checksum.update(result)
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi TypeError: Unicode-objects must be encoded before hashing
2019-01-16 16:37:06.597 50 ERROR glance.common.wsgi
2019-01-16 16:37:06.672 50 INFO eventlet.wsgi.server [req-4d3e351d-6cf8-499b-9e81-5534118e660f e45ef0ad944344f28c080a0a0ffafdea b6f89d6977da49db938448a470013256 - default default] 192.168.24.2 - - [16/Jan/2019 16:37:06] "PUT /v2/images/42d555ef-3757-4b6e-b91f-0969d6c8302a/file HTTP/1.1" 500 454 2.705659

Upstream bug seems to be https://bugs.launchpad.net/glance/+bug/1772018
The following review fixes it for us: https://review.openstack.org/#/c/620234/ but there seems to be some disagreement over the correctness of the fix.

The versions used where we saw this are:
python3-glanceclient-2.15.0-0.20181226111056.c4c92ec.fc28.noarch
python3-glance-17.1.0-0.20190109134015.fb2c2e7.fc28.noarch
openstack-glance-17.1.0-0.20190109134015.fb2c2e7.fc28.noarch
python3-glance-store-0.27.1-0.20190108172526.de27161.fc28.noarch

Comment 1 Cyril Roelandt 2019-02-19 17:11:31 UTC

OK, I'll keep an eye on the upstream fix. I think it should make its way into OSP15, so we may not even need to backport it.

Comment 2 Alexander Chuzhoy 2019-03-18 16:07:37 UTC

*** Bug 1689432 has been marked as a duplicate of this bug. ***

Comment 3 Alistair Tonner 2019-03-19 18:02:56 UTC

*** Bug 1689989 has been marked as a duplicate of this bug. ***

Comment 4 Brian Rosmaita 2019-03-20 12:05:34 UTC

The stable/stein branch of glance_store was already cut.  Upstream backport: https://review.openstack.org/#/c/644839/

Comment 15 errata-xmlrpc 2019-09-21 11:20:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811

Note You need to log in before you can comment on or make changes to this bug.