Bug 1023932 - swift: failure and no ability to reset contents for reupload after failing to create an image
swift: failure and no ability to reset contents for reupload after failing to...
Status: CLOSED DUPLICATE of bug 1012407
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-glance (Show other bugs)
4.0
x86_64 Linux
unspecified Severity medium
: ---
: 4.0
Assigned To: RHOS Maint
Haim
storage
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-28 07:27 EDT by Dafna Ron
Modified: 2016-04-26 10:22 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-11 08:58:07 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
glance api log (1.44 KB, application/x-xz)
2013-10-28 07:27 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1245444 None None None Never

  None (edit)
Description Dafna Ron 2013-10-28 07:27:45 EDT
Created attachment 816777 [details]
glance api log

Description of problem:

I configured swift as glance backend and fail to create an image due to storage space issues on data servers. 
after i fail to create an image I can see in swift list glance that we have some odd image leftovers and the glance log shows the following error: 

failure and no ability to reset contents for reupload

Version-Release number of selected component (if applicable):

openstack-swift-1.9.1-2.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. fail to create image due to disk space issues on data servers (you can create small size disks) 
2. configure swift as glance backend
3. create an image from horizon

Actual results:

I see "leftovers" in glance container and log shows the following: 

Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload.


Expected results:

we should clean container in case of failure 

Additional info:


2013-10-28 13:12:39.518 4957 ERROR glance.store.swift [-] Failed to add object to Swift.
Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload.
2013-10-28 13:12:39.518 4957 ERROR glance.api.v1.upload_utils [-] Failed to upload image eb51f83b-7993-4e94-bba3-9ad9dc7e8525
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.6/site-packages/glance/api/v1/upload_utils.py", line 101, in upload_data_to_store
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils     store)
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.6/site-packages/glance/store/__init__.py", line 333, in store_add_to_backend
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils     (location, size, checksum, metadata) = store.add(image_id, data, size)
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.6/site-packages/glance/store/swift.py", line 441, in add
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils     raise glance.store.BackendException(msg)
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils BackendException: Failed to add object to Swift.
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload.
2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils 


[root@opens-vdsb ~(keystone_glance)]# swift list glance 
5a2a41a3-cfb7-4ba6-80f0-3c43690cdd02-00001
be662602-52e9-49cc-93f7-955847190d76
eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001
Comment 1 Pete Zaitcev 2013-10-29 00:59:41 EDT
I do not understand why this was filed against the openstack-swift component.
It's not like Swift can do anything about it. Should I reassign it to Glance,
or I am missing something in the report?
Comment 2 Ayal Baron 2013-10-29 06:37:53 EDT
(In reply to Pete Zaitcev from comment #1)
> I do not understand why this was filed against the openstack-swift component.
> It's not like Swift can do anything about it. Should I reassign it to Glance,
> or I am missing something in the report?

2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001', ...) failure and no ability to reset contents for reupload.

?
Comment 3 Pete Zaitcev 2013-10-29 13:07:39 EDT
(In reply to Ayal Baron from comment #2)
> (In reply to Pete Zaitcev from comment #1)
> > I do not understand why this was filed against the openstack-swift component.
> > It's not like Swift can do anything about it.

> 2013-10-28 13:12:39.518 4957 TRACE glance.api.v1.upload_utils Got error from
> Swift: put_object('glance', 'eb51f83b-7993-4e94-bba3-9ad9dc7e8525-00001',
> ...) failure and no ability to reset contents for reupload.
> 
> ?

So, what is Swift supposed to do about this? Glance told it to store
an object, it reported that there was no space. Fine, now what? It cannot
know what Glance meant to do with a collection of objects. It is up to
Glance to delete remaining fragments.

This could be a bug if there was some controversy about Swift breaking
up objects or adding versioned objects or doing other things under the
API boundary, then not cleaning up. But in this case I do not see how
Swift can help to resolve the issue.
Comment 4 Ayal Baron 2013-10-30 11:39:24 EDT
Looks like the 'no ability to reset contents' error is coming from glance.  Swift is just reporting an error since there was not enough space.
Moving to glance
Comment 5 Flavio Percoco 2013-11-11 08:58:07 EST
This bug is a duplicate of #1012407, feel free to re-open it if you think otherwise.

*** This bug has been marked as a duplicate of bug 1012407 ***

Note You need to log in before you can comment on or make changes to this bug.