Bug 1614928 - Unable to upload OC images to glance: Faile d to upload image data due to internal error
Summary: Unable to upload OC images to glance: Faile d to upload image data due to int...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-glance-store
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 14.0 (Rocky)
Assignee: Cyril Roelandt
QA Contact: Filip Hubík
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-10 18:36 UTC by Alexander Chuzhoy
Modified: 2022-08-11 15:58 UTC (History)
14 users (show)

Fixed In Version: python-glance-store-0.26.1-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 11:51:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
RDO 15077 0 None None None 2018-08-10 19:36:15 UTC
Red Hat Issue Tracker OSP-5743 0 None None None 2022-08-11 15:58:02 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:51:23 UTC

Description Alexander Chuzhoy 2018-08-10 18:36:21 UTC
Unable to upload OC images to glance: Faile d to upload image data due to internal error

Environment:
instack-undercloud-9.2.1-0.20180803181448.be5fa97.el7ost.noarch
puppet-glance-13.2.0-0.20180728150233.a65de83.el7ost.noarch
python2-glanceclient-2.11.1-0.20180608145253.1cf00a5.el7ost.noarch


Steps to reproduce:
1. Deploy undercloud
2. Try to upload images to glance


Result:
(undercloud) [stack@undercloud ~]$ openstack overcloud image upload
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

Comment 3 Alexander Chuzhoy 2018-08-10 18:51:32 UTC
Errors in /var/log/containers/glance/api.log:

2018-08-10 14:36:11.104 26 ERROR glance.api.v2.image_data [req-cd51852d-7df0-47a1-8187-c6f7e71228a1 4c59164086924d5083d12cf70922679b e57037826de841c392b6cb7860452a93 - default default] Faile
d to upload image data due to internal error
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi [req-cd51852d-7df0-47a1-8187-c6f7e71228a1 4c59164086924d5083d12cf70922679b e57037826de841c392b6cb7860452a93 - default default] Caught erro
r: 'module' object has no attribute 'add_to_backend_with_multihash': AttributeError: 'module' object has no attribute 'add_to_backend_with_multihash'
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi Traceback (most recent call last):
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1274, in __call__
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     request, **action_args)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1317, in dispatch
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     return method(*args, **kwargs)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 417, in wrapped
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 279, in upload
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     self._restore(image_repo, image)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     self.force_reraise()
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 144, in upload
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 198, in set_data
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     self.base.set_data(data, size, backend=backend)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 485, in set_data
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     self.force_reraise()
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 432, in set_data
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     self.repo.set_data(data, size, backend=backend)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 193, in set_data
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 310, in set_data
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     self.image.set_data(data, size=size, backend=backend)
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/location.py", line 475, in set_data
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi     loc_meta) = self.store_api.add_to_backend_with_multihash(
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi AttributeError: 'module' object has no attribute 'add_to_backend_with_multihash'
2018-08-10 14:36:11.205 26 ERROR glance.common.wsgi

Comment 4 Jon Schlueter 2018-08-10 19:24:24 UTC
missing python-chardet-3.0.4-7.el7 that came along with python-requests-2.19.1-3.el7

https://review.rdoproject.org/r/#/c/15077/3/deps.yml

Comment 10 Erno Kuvaja 2018-08-14 13:57:56 UTC
Looks like we might be using old version of glance_store. Could someone please check which version of glance_store is included in these puddles for me?

Comment 12 Thierry Vignaud 2018-08-14 15:46:58 UTC
We've python-chardet-3.0.4-7.el7ost since 2018-08-13.1 puddle

Comment 13 Christian Schwede (cschwede) 2018-08-15 08:27:35 UTC
I had a look into the glance container, and it looks like we're missing an update of python-glance-store. This patch is missing: https://review.openstack.org/#/c/587098/
Thus we need an updated build of python-glance-store (latest is https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=732749).

Comment 15 Filip Hubík 2018-08-17 10:04:38 UTC
In freshest puddle (2018-08-15.1) passing UC install and image upload stages we have package python2-glance-store-0.26.1-1.el7ost present inside openstack-glance-api container.

Images are uploaded to glance registry and they are able to reach active state, they are not stuck in queued state anymore:

uc $ openstack image list
+--------------------------------------+------------------------+--------+
| ID                                   | Name                   | Status |
+--------------------------------------+------------------------+--------+
| 2c7f5391-c8b1-486b-b793-024d1b06ea8e | bm-deploy-kernel       | active |
| 7fb9e254-3348-41bd-bf2f-8c335bb166f8 | bm-deploy-ramdisk      | active |
| d37889df-ccc4-468d-9998-3dae962dc0df | overcloud-full         | active |
| e38d2bb7-ef2d-4f98-b2fc-32a780e566b5 | overcloud-full-initrd  | active |
| 4c5cad73-7b96-417f-8f70-0672ac8811fd | overcloud-full-vmlinuz | active |
+--------------------------------------+------------------------+--------+

Verified.

Comment 20 errata-xmlrpc 2019-01-11 11:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.