Bug 1039569

Summary: [Functional] Object Expirer doesn't clean up objects yet it accepts headers X-Delete-After & X-Delete-At
Product: Red Hat Gluster Storage Reporter: pushpesh sharma <psharma>
Component: gluster-swiftAssignee: Luis Pabón <lpabon>
Status: CLOSED ERRATA QA Contact: Bhaskar Bandari <bbandari>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.1CC: bbandari, david.macdonald, madam, ppai, psriniva, rhs-bugs, ssamanta, vagarwal
Target Milestone: ---   
Target Release: RHGS 3.0.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, headers X-Delete-At and X-Delete-After were accepted although object expiration feature was not fully implemented, thus leading to confusion. With this fix, the X-Delete-At and X-Delete-After headers are not accepted.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-22 19:30:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description pushpesh sharma 2013-12-09 13:48:36 UTC 
Description of problem:

A 404 is returned for a GET on an expired object. As "object-expirer" daemon is not run in gluster-swift environment, the files however are not deleted from gluster volume.

Version-Release number of selected component (if applicable):
RHS2.1-GA & RHS2.1-U1

How reproducible:


Steps to Reproduce:
Few study updates: 
Gluster-Swift does not support object expiration yet.

Object expiration is a two fold task.
1) Store the expiration time to the object metadata. Return 404 when such time elapsed.
2) special service "object-expirer" should run to delete such objects and their metadata <wherever stored>.

Gluster-Swift currently does not store X-Delete-At or X-Delete-After header in it's xattr. It has no code to perform cross-checks with current epoch and header's mentioned epoch to return 404 if it has elapsed. This is verified and failed with latest Gluster-Swift code. So this #1 can be implemented in Gluster-Swift with some code changes. Needs more investigation.

2) Interesting thing is in plain swift environment, Openstack Swift stores expiration related metadata of those objects in a special account. SWAuth could be useful in here when we maintain multiple accounts per volume.

Openstack Swift's object expirer does walk across such metadata and removes those objects internally.

Current account = volume would not work in there. It would need multiple accounts per volume which SWAuth can privide. Just a suggestion. Need to investigate on this further.


The "object-expirer" daemon, as of now, depends on a separate special account that keeps track of objects to be deleted. As accounts=volume in gluster-swift, we may have to wait till multiple accounts per volume are implemented OR we could investigate overriding some parts of object-expirer if we want the feature sooner.

Actual results:


Expected results:


Additional info:
Comment 2 Luis Pabón 2013-12-12 03:47:12 UTC 
New fix does not allow X-Delete-After or X-Delete-At headers to be used in POST and PUT commands.  The fix has been merged into the havana branch:

http://review.gluster.org/#/c/6444/
Comment 3 Luis Pabón 2013-12-12 03:48:04 UTC 
Upstream RPMs available here:
https://launchpad.net/gluster-swift/havana/1.10.0-1
Comment 4 pushpesh sharma 2014-06-06 06:48:07 UTC 
]# curl -i -v -X PUT -T anaconda-ks.cfg -H 'X-Delete-After: 20' http://10.65.202.205:8080/v1/AUTH_test/cont/obj-exp 
* About to connect() to 10.65.202.205 port 8080 (#0)
*   Trying 10.65.202.205... connected
* Connected to 10.65.202.205 (10.65.202.205) port 8080 (#0)
> PUT /v1/AUTH_test/cont/obj-exp HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.65.202.205:8080
> Accept: */*
> X-Delete-After: 20
> Content-Length: 1190
> Expect: 100-continue
> 
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Length: 52
Content-Length: 52
< Content-Type: text/plain
Content-Type: text/plain
< X-Trans-Id: tx7623e5c3b3b84071a6178-005391b417
X-Trans-Id: tx7623e5c3b3b84071a6178-005391b417
< Date: Fri, 06 Jun 2014 12:29:11 GMT
Date: Fri, 06 Jun 2014 12:29:11 GMT

< 
* Connection #0 to host 10.65.202.205 left intact
* Closing connection #0
x-delete-at,x-delete-after headers are not supported[root@RHSS2 ~]# 
[root@RHSS2 ~]# curl -i -v -X PUT -T anaconda-ks.cfg -H 'X-Delete-At: 87920' http://10.65.202.205:8080/v1/AUTH_test/cont/obj-exp 
* About to connect() to 10.65.202.205 port 8080 (#0)
*   Trying 10.65.202.205... connected
* Connected to 10.65.202.205 (10.65.202.205) port 8080 (#0)
> PUT /v1/AUTH_test/cont/obj-exp HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.65.202.205:8080
> Accept: */*
> X-Delete-At: 87920
> Content-Length: 1190
> Expect: 100-continue
> 
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Length: 52
Content-Length: 52
< Content-Type: text/plain
Content-Type: text/plain
< X-Trans-Id: txb58a0e573ab249718a5fc-005391b432
X-Trans-Id: txb58a0e573ab249718a5fc-005391b432
< Date: Fri, 06 Jun 2014 12:29:38 GMT
Date: Fri, 06 Jun 2014 12:29:38 GMT

< 
* Connection #0 to host 10.65.202.205 left intact
* Closing connection #0
x-delete-at,x-delete-after headers are not supported[root@RHSS2 ~]# 
[root@RHSS2 ~]# 


Verified On:-
#rpm -qa|grep swift
openstack-swift-account-1.10.0-3.el6ost.noarch
openstack-swift-object-1.10.0-3.el6ost.noarch
swiftonfile-1.10.0-6.el6rhs.noarch
openstack-swift-plugin-swift3-1.0.0-0.20120711git.1.el6ost.noarch
openstack-swift-proxy-1.10.0-3.el6ost.noarch
python-swiftclient-1.8.0-1.el6ost.noarch
openstack-swift-container-1.10.0-3.el6ost.noarch
openstack-swift-1.10.0-3.el6ost.noarch
Comment 5 Prashanth Pai 2014-06-18 10:34:57 UTC 
Updated doc text accordingly.
Comment 6 Pavithra 2014-07-23 10:06:56 UTC 
Hi ppai,

Please review the doc text for technical accuracy and sign off.
Comment 7 Prashanth Pai 2014-07-23 10:19:26 UTC 
The doc text meant slightly different from the actual bug fix. Have updated the same.
Comment 11 errata-xmlrpc 2014-09-22 19:30:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1278.html