Description of problem: Cannot read old objects that were created prior to hammer if these were uploaded through the S3 multipart upload, and if they start with underscore. Issue is with the tail objects that are assigned an object locator that doesn't match the locator that is created in older version. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. Create multipart object starting with underscore on firefly (1.2.3) 2. Upgrade to hammer (1.3.0) 3. Try to read it. Actual results: cannot read object.
Just to add more info which Yehuda mentioned in irc if we dont take this in 1.3.0, we will hit the issue when we upgrade to 1.3.1 or any async. so the second case where we can hit the above issue is (1) fresh install of 1.3.0 (multipart objs with underscore) -> 1.3.1 or async
will run radosgw teuthology tests on 1.3.0(rpm) which yehuda ran on upstream
Yehuda, I was able to access the objects with underscore., but post upgrade(0.94.1.13) when I ran the script which I was using to get the bucket list and downloading Objfiles,. I am seeing traceback error.. The same script worked fine before upgrading(0.80.8.7) Traceback (most recent call last): File "./s3test.py", line 21, in <module> for key in bucket.list(): File "/usr/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister encoding_type=encoding_type) File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 472, in get_all_keys '', headers, **params) File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 410, in _get_all response.status, response.reason, body) boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code></Error> I have pasted the execution logs which I performed to verify this bug.. http://pastebin.test.redhat.com/291682
It seems that you get that on what seems like to be an unrelated bucket (my-new-bucket), so you might have found some other unrelated issue. The specific multipart issue seems to be working ok. Note that you didn't upgrade (or restart) all your osds after the upgrade, it is likely that the issue you're seeing is related to that: [root@magna110 ceph]# for i in `seq 0 7`; do ceph tell osd.$i version; done { "version": "ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)" } { "version": "ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)" } { "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"} { "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"} { "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"} { "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"} { "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"} { "version": "ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)"}
Thanks Yehuda .. I missed restarting the ceph osd's.. My bad.. I just checked the ceph --version on all the Nodes and it did show as 0.94.1.. Anyways, I restarted the OSD's and it seems to be working now.. ----------------------------------------------------------------------- [root@magna110 ceph]# ./s3test.py bigbucket 2015-06-20T10:56:10.000Z bucketbig 2015-06-19T19:51:23.000Z ....@_@_@_.... 104857603 2015-06-20T11:18:55.000Z DONE======== @@@@@@__obJ______1__1 100663296 2015-06-20T10:42:03.000Z DONE======== @_@__4underscore____@_@_ 104857603 2015-06-20T11:18:06.000Z DONE======== ___3underscores___ 100663296 2015-06-20T11:13:17.000Z DONE======== ____4underscore____ 104857603 2015-06-20T11:16:32.000Z DONE======== _____Hello_hyelloji@@@@.... 104857603 2015-06-20T11:21:01.000Z DONE======== __________________ 104857603 2015-06-20T11:19:36.000Z DONE======== __________________Test@@@@.... 104857603 2015-06-20T11:19:54.000Z DONE======== _______________________________@@@@@@__ 100663296 2015-06-20T10:42:37.000Z DONE======== _______________________________@@@@@@__obJ______1__1 100663296 2015-06-20T10:42:21.000Z DONE======== __obJ______1__1 100663296 2015-06-20T10:41:35.000Z DONE======== big.dat 104857600 2015-06-20T07:40:01.000Z DONE======== my-new-bucket 2015-06-19T19:48:44.000Z [root@magna110 ceph]# Moving this to verified State..
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1183