Bug 1258617

Summary: rgw: COPYing an old object onto itself produces a truncated object
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Yehuda Sadeh <yehuda>
Component: RGWAssignee: Ken Dreyer (Red Hat) <kdreyer>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 1.2.3CC: cbodley, ceph-eng-bugs, kbader, kdreyer, mbenjamin, nlevine, owasserm, sweil, tmuthami
Target Milestone: rc   
Target Release: 1.2.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-0.80.8-17 (RHEL), ceph v0.80.8.5 (Ubuntu & CentOS) Doc Type: Bug Fix
Doc Text:
Prior to this update, COPYing an object onto itself (e.g., to change its metadata) reliably produces a truncated object, if the source object was originally created with rgw older than RHCS 1.2 using a non-multi-part upload. If the object was originally created via a multi-part upload, or was created with a newer rgw, such as RHCS 1.2, the problem does not occur. The truncated object has correct metadata, including the original size, but the underlying RADOS object is smaller. When a client attempts to fetch the object, it receives less data than indicated by the Content-Length, blocks for more, and eventually times out. RGW has been updated to handle old manifests properly, and COPYing old objects in the above scenario works as expected.
 Story Points: ---
Clone Of:
: 1258618 (view as bug list) Environment:
Last Closed: 2015-10-01 20:25:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yehuda Sadeh 2015-08-31 19:16:26 UTC 
Description of problem:
On at least Giant and Hammer, COPYing an object onto itself (e.g., to change its metadata) reliably produces a truncated object, if the source object was originally created with rgw older than firefly using a non-multi-part upload. If the object was originally created via a multi-part upload, or was created with a newer rgw (tested with Giant and Hammer), the problem does not occur.

The truncated object has correct metadata, including the original size, but the underlying RADOS object is smaller. When a client attempts to fetch the object, it receives less data than indicated by the Content-Length, blocks for more, and eventually times out.


How reproducible:
always

Steps to Reproduce:
1. Install dumpling
2. Create object > 512k
3. Upgrade
4. Copy object into itself (modify attributes)

Actual results:
Object cannot be read successfully


Expected results:
Object should be read successfully

Additional info:
Comment 3 Ken Dreyer (Red Hat) 2015-09-15 17:09:10 UTC 
We will take this fix in v1.2.3.2 on non-RHEL, and fix it via a new async update for RHEL.
Comment 4 Ken Dreyer (Red Hat) 2015-09-15 21:58:12 UTC 
Fix will be in non-RHEL Ceph v0.80.8.5
Comment 9 Tamil 2015-09-30 23:30:09 UTC 
the fix is verified on rhel 6.7 and it works like a charm!!

steps to reproduce the bug:

1. install 1.1 on rhel 6.7, install and configure rgw on rhel 6.7
2. create a bucket and upload few objects, each object > 512 kb
i used swift commands,
dd if=/dev/zero of=./data bs=1024 count=102400
swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" upload my-new-bucket data
3. upgrade to 1.2.3
4. install mod_proxy_fcgi and reconfigure rgw to work with 1.2.3
5. restart rgw and reproduce the bug
6. inorder to reproduce the bug, copy an object that you created in 1.1 onto itself and verify if the object is truncated,
curl -i -X PUT  -H "X-Auth-Token:<AUTH_TOKEN>" http://10.8.128.93/swift/v1/my-new-bucket/data -H "X-Copy-From: /my-new-bucket1/data" 
swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" download my-new-bucket data
check for the size of the data, it should be truncated.
7. update to the puddle builds from the errata to verify the bug fix.
8. pick the object [other than the one that is already truncated ], redo the commands on step 6 [replace the object in step6 with the one you picked] and verify the data is not truncated.

[ubuntu@magna093 ~]$ curl -i -X PUT  -H "X-Auth-Token:AUTH_rgwtk0e00000074657374757365723a7377696674b103b727df72eba8d7bd0d5600617f38ffb882c684b1b0b31ff673a31bf27501ff45cc70" http://10.8.128.93/swift/v1/my-new-bucket1/data1 -H "X-Copy-From: /my-new-bucket1/data1"
HTTP/1.1 201 Created
Date: Wed, 30 Sep 2015 23:13:16 GMT
Content-Length: 0
Connection: close
Content-Type: text/plain; charset=utf-8

[ubuntu@magna093 ~]$ swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" download my-new-bucket1 data1
data1 [auth 0.031s, headers 0.046s, total 0.538s, 206.910 MB/s]
[ubuntu@magna093 ~]$ ls -l
total 316432
-rw-r--r--. 1 ubuntu kvm 104333312 Sep 30 17:33 data
-rw-r--r--. 1 ubuntu kvm 104857600 Sep 30 17:33 data1
Comment 10 Tamil 2015-10-01 16:20:22 UTC 
fix verified on rhel 7.1 as well.
Comment 11 errata-xmlrpc 2015-10-01 20:25:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1856.html