Bug 1258617 - rgw: COPYing an old object onto itself produces a truncated object
rgw: COPYing an old object onto itself produces a truncated object
Status: CLOSED ERRATA
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
1.2.3
Unspecified Unspecified
unspecified Severity unspecified
: rc
: 1.2.4
Assigned To: Ken Dreyer (Red Hat)
ceph-qe-bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-31 15:16 EDT by Yehuda Sadeh
Modified: 2017-07-30 12:05 EDT (History)
9 users (show)

See Also:
Fixed In Version: ceph-0.80.8-17 (RHEL), ceph v0.80.8.5 (Ubuntu & CentOS)
Doc Type: Bug Fix
Doc Text:
Prior to this update, COPYing an object onto itself (e.g., to change its metadata) reliably produces a truncated object, if the source object was originally created with rgw older than RHCS 1.2 using a non-multi-part upload. If the object was originally created via a multi-part upload, or was created with a newer rgw, such as RHCS 1.2, the problem does not occur. The truncated object has correct metadata, including the original size, but the underlying RADOS object is smaller. When a client attempts to fetch the object, it receives less data than indicated by the Content-Length, blocks for more, and eventually times out. RGW has been updated to handle old manifests properly, and COPYing old objects in the above scenario works as expected.
Story Points: ---
Clone Of:
: 1258618 (view as bug list)
Environment:
Last Closed: 2015-10-01 16:25:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 11455 None None None Never

  None (edit)
Description Yehuda Sadeh 2015-08-31 15:16:26 EDT
Description of problem:
On at least Giant and Hammer, COPYing an object onto itself (e.g., to change its metadata) reliably produces a truncated object, if the source object was originally created with rgw older than firefly using a non-multi-part upload. If the object was originally created via a multi-part upload, or was created with a newer rgw (tested with Giant and Hammer), the problem does not occur.

The truncated object has correct metadata, including the original size, but the underlying RADOS object is smaller. When a client attempts to fetch the object, it receives less data than indicated by the Content-Length, blocks for more, and eventually times out.


How reproducible:
always

Steps to Reproduce:
1. Install dumpling
2. Create object > 512k
3. Upgrade
4. Copy object into itself (modify attributes)

Actual results:
Object cannot be read successfully


Expected results:
Object should be read successfully

Additional info:
Comment 3 Ken Dreyer (Red Hat) 2015-09-15 13:09:10 EDT
We will take this fix in v1.2.3.2 on non-RHEL, and fix it via a new async update for RHEL.
Comment 4 Ken Dreyer (Red Hat) 2015-09-15 17:58:12 EDT
Fix will be in non-RHEL Ceph v0.80.8.5
Comment 9 Tamil 2015-09-30 19:30:09 EDT
the fix is verified on rhel 6.7 and it works like a charm!!

steps to reproduce the bug:

1. install 1.1 on rhel 6.7, install and configure rgw on rhel 6.7
2. create a bucket and upload few objects, each object > 512 kb
i used swift commands,
dd if=/dev/zero of=./data bs=1024 count=102400
swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" upload my-new-bucket data
3. upgrade to 1.2.3
4. install mod_proxy_fcgi and reconfigure rgw to work with 1.2.3
5. restart rgw and reproduce the bug
6. inorder to reproduce the bug, copy an object that you created in 1.1 onto itself and verify if the object is truncated,
curl -i -X PUT  -H "X-Auth-Token:<AUTH_TOKEN>" http://10.8.128.93/swift/v1/my-new-bucket/data -H "X-Copy-From: /my-new-bucket1/data" 
swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" download my-new-bucket data
check for the size of the data, it should be truncated.
7. update to the puddle builds from the errata to verify the bug fix.
8. pick the object [other than the one that is already truncated ], redo the commands on step 6 [replace the object in step6 with the one you picked] and verify the data is not truncated.

[ubuntu@magna093 ~]$ curl -i -X PUT  -H "X-Auth-Token:AUTH_rgwtk0e00000074657374757365723a7377696674b103b727df72eba8d7bd0d5600617f38ffb882c684b1b0b31ff673a31bf27501ff45cc70" http://10.8.128.93/swift/v1/my-new-bucket1/data1 -H "X-Copy-From: /my-new-bucket1/data1"
HTTP/1.1 201 Created
Date: Wed, 30 Sep 2015 23:13:16 GMT
Content-Length: 0
Connection: close
Content-Type: text/plain; charset=utf-8

[ubuntu@magna093 ~]$ swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" download my-new-bucket1 data1
data1 [auth 0.031s, headers 0.046s, total 0.538s, 206.910 MB/s]
[ubuntu@magna093 ~]$ ls -l
total 316432
-rw-r--r--. 1 ubuntu kvm 104333312 Sep 30 17:33 data
-rw-r--r--. 1 ubuntu kvm 104857600 Sep 30 17:33 data1
Comment 10 Tamil 2015-10-01 12:20:22 EDT
fix verified on rhel 7.1 as well.
Comment 11 errata-xmlrpc 2015-10-01 16:25:57 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1856.html

Note You need to log in before you can comment on or make changes to this bug.