Bug 1232953

Summary:	rgw: multipart objects starting with underscore are incompatible with older versions
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Yehuda Sadeh <yehuda>
Component:	RGW	Assignee:	Yehuda Sadeh <yehuda>
Status:	CLOSED ERRATA	QA Contact:	Tamil <tmuthami>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	1.3.0	CC:	branto, cbodley, ceph-eng-bugs, flucifre, hyelloji, icolle, kbader, mbenjamin, owasserm, sweil, tmuthami, vakulkar, yehuda
Target Milestone:	rc
Target Release:	1.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-0.94.1-13.el7cp	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-06-24 15:53:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yehuda Sadeh 2015-06-17 22:24:02 UTC

Description of problem:

Cannot read old objects that were created prior to hammer if these were uploaded through the S3 multipart upload, and if they start with underscore.

Issue is with the tail objects that are assigned an object locator that doesn't match the locator that is created in older version.



Version-Release number of selected component (if applicable):


How reproducible:
always


Steps to Reproduce:
1. Create multipart object starting with underscore on firefly (1.2.3)
2. Upgrade to hammer (1.3.0)
3. Try to read it.

Actual results:
cannot read object.

Comment 2 Vasu Kulkarni 2015-06-18 00:11:31 UTC

Just to add more info which Yehuda mentioned in irc

if we dont take this in 1.3.0,  we will hit the issue when we upgrade to 1.3.1
or any async.

so the second case where we can hit the above issue is

(1) fresh install of 1.3.0 (multipart objs with underscore) -> 1.3.1 or async

Comment 3 Vasu Kulkarni 2015-06-18 16:06:39 UTC

will run radosgw teuthology tests on 1.3.0(rpm) which yehuda ran on upstream

Comment 5 Hemanth Kumar 2015-06-20 16:24:10 UTC

Yehuda,

I was able to access the objects with underscore., but post upgrade(0.94.1.13) when I ran the script which I was using to get the bucket list and downloading Objfiles,. I am seeing traceback error.. The same script worked fine before upgrading(0.80.8.7)

Traceback (most recent call last):
  File "./s3test.py", line 21, in <module>
    for key in bucket.list():
  File "/usr/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister
    encoding_type=encoding_type)
  File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 472, in get_all_keys
    '', headers, **params)
  File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 410, in _get_all
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code></Error>



I have pasted the execution logs which I performed to verify this bug..
http://pastebin.test.redhat.com/291682

Comment 6 Yehuda Sadeh 2015-06-21 00:51:17 UTC

It seems that you get that on what seems like to be an unrelated bucket (my-new-bucket), so you might have found some other unrelated issue. The specific multipart issue seems to be working ok.

Note that you didn't upgrade (or restart) all your osds after the upgrade, it is likely that the issue you're seeing is related to that:

[root@magna110 ceph]# for i in `seq 0 7`; do ceph tell osd.$i version; done
{
    "version": "ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)"
}
{
    "version": "ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)"
}
{ "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}
{ "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}
{ "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}
{ "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}
{ "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}
{ "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}

Comment 7 Hemanth Kumar 2015-06-21 10:28:56 UTC

Thanks Yehuda ..
I missed restarting the ceph osd's.. My bad.. I just checked the ceph --version on all the Nodes and it did show as 0.94.1..

Anyways, I restarted the OSD's and it seems to be working now..
-----------------------------------------------------------------------

[root@magna110 ceph]# ./s3test.py 

bigbucket	2015-06-20T10:56:10.000Z
bucketbig	2015-06-19T19:51:23.000Z
....@_@_@_....	104857603	2015-06-20T11:18:55.000Z
DONE========
@@@@@@__obJ______1__1	100663296	2015-06-20T10:42:03.000Z
DONE========
@_@__4underscore____@_@_	104857603	2015-06-20T11:18:06.000Z
DONE========
___3underscores___	100663296	2015-06-20T11:13:17.000Z
DONE========
____4underscore____	104857603	2015-06-20T11:16:32.000Z
DONE========
_____Hello_hyelloji@@@@....	104857603	2015-06-20T11:21:01.000Z
DONE========
__________________	104857603	2015-06-20T11:19:36.000Z
DONE========
__________________Test@@@@....	104857603	2015-06-20T11:19:54.000Z
DONE========
_______________________________@@@@@@__	100663296	2015-06-20T10:42:37.000Z
DONE========
_______________________________@@@@@@__obJ______1__1	100663296	2015-06-20T10:42:21.000Z
DONE========
__obJ______1__1	100663296	2015-06-20T10:41:35.000Z
DONE========
big.dat	104857600	2015-06-20T07:40:01.000Z
DONE========
my-new-bucket	2015-06-19T19:48:44.000Z
[root@magna110 ceph]# 



Moving this to verified State..

Comment 9 errata-xmlrpc 2015-06-24 15:53:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1183