Red Hat Bugzilla – Bug 1023932
swift: failure and no ability to reset contents for reupload after failing to create an image
Last modified: 2016-04-26 10:22:26 EDT
Created attachment 816777 [details] glance api log Description of problem: I configured swift as glance backend and fail to create an image due to storage space issues on data servers. after i fail to create an image I can see in swift list glance that we have some odd image leftovers and the glance log shows the following error: failure and no ability to reset contents for reupload Version-Release number of selected component (if applicable): openstack-swift-1.9.1-2.el6ost.noarch How reproducible: 100% Steps to Reproduce: 1. fail to create image due to disk space issues on data servers (you can create small size disks) 2. configure swift as glance backend 3. create an image from horizon Actual results: I see "leftovers" in glance container and log shows the following: Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload. Expected results: we should clean container in case of failure Additional info: 2013-10-28 13:12:39.518 4957 ERROR glance.store.swift [-] Failed to add object to Swift. Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload. 2013-10-28 13:12:39.518 4957 ERROR glance.api.v1.upload_utils [-] Failed to upload image eb51f83b-7993-4e94-bba3-9ad9dc7e8525 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Traceback (most recent call last): 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 101, in upload_data_to_store 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils store) 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/glance/store/__init__.py", line 333, in store_add_to_backend 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size) 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.6/site-packages/glance/store/swift.py", line 441, in add 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils raise glance.store.BackendException(msg) 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils BackendException: Failed to add object to Swift. 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload. 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils [root@opens-vdsb ~(keystone_glance)]# swift list glance 5a2a41a3-cfb7-4ba6-80f0-3c43690cdd02-00001 be662602-52e9-49cc-93f7-955847190d76 eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001
I do not understand why this was filed against the openstack-swift component. It's not like Swift can do anything about it. Should I reassign it to Glance, or I am missing something in the report?
(In reply to Pete Zaitcev from comment #1) > I do not understand why this was filed against the openstack-swift component. > It's not like Swift can do anything about it. Should I reassign it to Glance, > or I am missing something in the report? 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload. ?
(In reply to Ayal Baron from comment #2) > (In reply to Pete Zaitcev from comment #1) > > I do not understand why this was filed against the openstack-swift component. > > It's not like Swift can do anything about it. > 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Got error from > Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', > ...) failure and no ability to reset contents for reupload. > > ? So, what is Swift supposed to do about this? Glance told it to store an object, it reported that there was no space. Fine, now what? It cannot know what Glance meant to do with a collection of objects. It is up to Glance to delete remaining fragments. This could be a bug if there was some controversy about Swift breaking up objects or adding versioned objects or doing other things under the API boundary, then not cleaning up. But in this case I do not see how Swift can help to resolve the issue.
Looks like the 'no ability to reset contents' error is coming from glance. Swift is just reporting an error since there was not enough space. Moving to glance
This bug is a duplicate of #1012407, feel free to re-open it if you think otherwise. *** This bug has been marked as a duplicate of bug 1012407 ***