Bug 1717962 - Failed to upload image data due to internal error: OSError: unexpected end of file while parsing chunked data
Summary: Failed to upload image data due to internal error: OSError: unexpected end of...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-glance
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Cyril Roelandt
QA Contact: Mike Abrams
Tana
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-06 14:26 UTC by Alexander Chuzhoy
Modified: 2020-02-27 21:35 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-16 15:17:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Alexander Chuzhoy 2019-06-06 14:26:52 UTC
Environment:
openstack-glance-18.0.1-0.20190420013840.f772321.el8ost.noarch
puppet-glance-14.4.1-0.20190420122113.e1c293d.el8ost.noarch
python3-glance-18.0.1-0.20190420013840.f772321.el8ost.noarch
python3-glance-store-0.28.1-0.20190424191509.ff53fe3.el8ost.noarch
python3-glanceclient-2.16.0-0.20190312083532.44a4dbd.el8ost.noarch
openstack-tripleo-heat-templates-10.5.1-0.20190528190414.f41e7e4.1.el8ost.noarch
python3-tripleoclient-11.4.1-0.20190524190411.c67f486.el8ost.noarch

Steps to reproduce:
1. Deployed OC with ironic enabled.
2. Launches BM instances in OC (used overcloud-full images)
3. Attempted to run tempest.

Result:
Running tempest fails as the cirros image doesn't become active.
The following error was encountered in glance.


2019-06-06 05:07:12.782 24 ERROR glance.api.v2.image_data [req-21f35b24-af79-4df8-bd66-3a7837982a1a 5d58cefcddc940f1955cdf94f24397f8 b7512774ab4747908b4a9045307d74ad - default default] Failed to upload image data due to internal error: OSError: unexpected end of file while parsing chunked data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi [req-21f35b24-af79-4df8-bd66-3a7837982a1a 5d58cefcddc940f1955cdf94f24397f8 b7512774ab4747908b4a9045307d74ad - default default] Caught error: unexpected end of file while parsing chunked data: OSError: unexpected end of file while parsing chunked data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi Traceback (most recent call last):
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1427, in __call__
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     request, **action_args)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1470, in dispatch
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     return method(*args, **kwargs)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 417, in wrapped
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 290, in upload
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     self._restore(image_repo, image)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     self.force_reraise()
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     raise value
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 155, in upload
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 198, in set_data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     self.base.set_data(data, size, backend=backend)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 485, in set_data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     self.force_reraise()
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     raise value
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 432, in set_data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     self.repo.set_data(data, size, backend=backend)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 202, in set_data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 318, in set_data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     self.image.set_data(data, size=size, backend=backend)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/location.py", line 483, in set_data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     verifier=verifier)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/backend.py", line 541, in add_to_backend_with_multihash
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     image_id, data, size, hashing_algo, store, context, verifier)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/backend.py", line 518, in store_add_to_backend_with_multihash
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     image_id, data, size, hashing_algo, context=context, verifier=verifier)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 274, in add_adapter
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     metadata_dict) = store_add_fun(*args, **kwargs)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 225, in op_checker
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/_drivers/rbd.py", line 559, in add
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     raise exc
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/_drivers/rbd.py", line 507, in add
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     for chunk in chunks:
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance_store/common/utils.py", line 67, in chunkiter
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     chunk = fp.read(chunk_size)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 295, in read
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     result = self.data.read(i)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 122, in readfn
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     result = fd.read(*args)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 217, in read
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     return self._chunked_read(self.rfile, length)
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi   File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 188, in _chunked_read
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi     raise IOError("unexpected end of file while parsing chunked data")
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi OSError: unexpected end of file while parsing chunked data
2019-06-06 05:07:12.829 24 ERROR glance.common.wsgi 
BrokenPipeError: [Errno 32] Broken pipe

Comment 3 John Fulton 2019-06-11 15:11:45 UTC
Do you have this patch https://review.opendev.org/#/c/653486 related to chunked uploads and Ceph Nautilus?

Comment 9 Alexander Chuzhoy 2019-07-16 15:17:18 UTC
The issue doesn't reproduce for me.

Comment 10 Chris Janiszewski 2020-02-27 21:35:57 UTC
I just hit the same looking issue in OSP16 GA .. right after deployment my glance would not allow me to upload images. Rebooting control plane solved the error and I no longer see it.


Note You need to log in before you can comment on or make changes to this bug.