Description of problem: On at least Giant and Hammer, COPYing an object onto itself (e.g., to change its metadata) reliably produces a truncated object, if the source object was originally created with rgw older than firefly using a non-multi-part upload. If the object was originally created via a multi-part upload, or was created with a newer rgw (tested with Giant and Hammer), the problem does not occur. The truncated object has correct metadata, including the original size, but the underlying RADOS object is smaller. When a client attempts to fetch the object, it receives less data than indicated by the Content-Length, blocks for more, and eventually times out. How reproducible: always Steps to Reproduce: 1. Install dumpling 2. Create object > 512k 3. Upgrade 4. Copy object into itself (modify attributes) Actual results: Object cannot be read successfully Expected results: Object should be read successfully Additional info:
We will take this fix in v1.2.3.2 on non-RHEL, and fix it via a new async update for RHEL.
Fix will be in non-RHEL Ceph v0.80.8.5
the fix is verified on rhel 6.7 and it works like a charm!! steps to reproduce the bug: 1. install 1.1 on rhel 6.7, install and configure rgw on rhel 6.7 2. create a bucket and upload few objects, each object > 512 kb i used swift commands, dd if=/dev/zero of=./data bs=1024 count=102400 swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" upload my-new-bucket data 3. upgrade to 1.2.3 4. install mod_proxy_fcgi and reconfigure rgw to work with 1.2.3 5. restart rgw and reproduce the bug 6. inorder to reproduce the bug, copy an object that you created in 1.1 onto itself and verify if the object is truncated, curl -i -X PUT -H "X-Auth-Token:<AUTH_TOKEN>" http://10.8.128.93/swift/v1/my-new-bucket/data -H "X-Copy-From: /my-new-bucket1/data" swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" download my-new-bucket data check for the size of the data, it should be truncated. 7. update to the puddle builds from the errata to verify the bug fix. 8. pick the object [other than the one that is already truncated ], redo the commands on step 6 [replace the object in step6 with the one you picked] and verify the data is not truncated. [ubuntu@magna093 ~]$ curl -i -X PUT -H "X-Auth-Token:AUTH_rgwtk0e00000074657374757365723a7377696674b103b727df72eba8d7bd0d5600617f38ffb882c684b1b0b31ff673a31bf27501ff45cc70" http://10.8.128.93/swift/v1/my-new-bucket1/data1 -H "X-Copy-From: /my-new-bucket1/data1" HTTP/1.1 201 Created Date: Wed, 30 Sep 2015 23:13:16 GMT Content-Length: 0 Connection: close Content-Type: text/plain; charset=utf-8 [ubuntu@magna093 ~]$ swift -A http://10.8.128.93/auth/1.0 -U testuser:swift -K "AD/TpoPpOy7lC7C6o6ioXHV/I9jB3Vj3LCzUycFF" download my-new-bucket1 data1 data1 [auth 0.031s, headers 0.046s, total 0.538s, 206.910 MB/s] [ubuntu@magna093 ~]$ ls -l total 316432 -rw-r--r--. 1 ubuntu kvm 104333312 Sep 30 17:33 data -rw-r--r--. 1 ubuntu kvm 104857600 Sep 30 17:33 data1
fix verified on rhel 7.1 as well.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1856.html