Bug 1584763 - Some versioned objects don't sync when uploaded with 's3cmd sync'
Summary: Some versioned objects don't sync when uploaded with 's3cmd sync'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW-Multisite
Version: 2.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z1
: 2.5
Assignee: Casey Bodley
QA Contact: Tejas
Aron Gunn
URL:
Whiteboard:
Depends On:
Blocks: 1585238 1585239
TreeView+ depends on / blocked
 
Reported: 2018-05-31 15:16 UTC by Casey Bodley
Modified: 2018-07-26 18:08 UTC (History)
7 users (show)

Fixed In Version: RHEL: ceph-10.2.10-27.el7cp Ubuntu: ceph_10.2.10-22redhat1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1585238 1585239 (view as bug list)
Environment:
Last Closed: 2018-07-26 18:06:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2261 None None None 2018-07-26 18:08:06 UTC
Ceph Project Bug Tracker 24367 None None None 2018-05-31 18:23:20 UTC

Description Casey Bodley 2018-05-31 15:16:40 UTC
Description of problem:

's3cmd sync' has some logic that compares the md5sum of local objects, and uses COPY for matching objects rather than uploading them separately with PUT. After COPYing objects, it then sets ACLs on the copied objects as a separate operation.

This second PutACL operation is treated as a separate transaction on the bucket index with a newer timestamp, so multisite sync chooses this operation as the most recent version to apply. However, this bucket index operation does not carry all of the necessary information related to object versioning - this information is only present in the 'link_olh' operation, which is only generated by the initial PUT or COPY.

As a result, when sync tries to process this PutACL operation, it skips the object with a message like:

> bucket skipping sync obj: versioned object will be synced on link_olh


Version-Release number of selected component (if applicable):

All versions of RHCS.


How reproducible:

Fairly reproduceable for objects uploaded with COPY. 


Steps to Reproduce:

1. Deploy a basic multisite configuration with two zones.
2. Create a BUCKET on the primary zone.
3. Enable versioning on the BUCKET.
4. Upload a single dummy object to the BUCKET. (this will trigger a full sync so that later objects are processed during incremental sync)
5. Create a local directory with 10 identical files:

$ mkdir versiontest
$ for i in `seq 0 9` ; do echo hello > versiontest/file$i ; done

6. Sync this directory into the bucket:

$ s3cmd sync versiontest s3://BUCKET

7. Let the bucket sync, then list the bucket on both zones.


Actual results:

Some objects are not present on the secondary zone.

primary zone: $ s3cmd ls -r s3://BUCKET
2018-05-31 15:08        41   s3://BUCKET/dummy
2018-05-31 15:08         6   s3://BUCKET/versiontest/file1
2018-05-31 15:08         6   s3://BUCKET/versiontest/file10
2018-05-31 15:08         6   s3://BUCKET/versiontest/file2
2018-05-31 15:08         6   s3://BUCKET/versiontest/file3
2018-05-31 15:08         6   s3://BUCKET/versiontest/file4
2018-05-31 15:08         6   s3://BUCKET/versiontest/file5
2018-05-31 15:08         6   s3://BUCKET/versiontest/file6
2018-05-31 15:08         6   s3://BUCKET/versiontest/file7
2018-05-31 15:08         6   s3://BUCKET/versiontest/file8
2018-05-31 15:08         6   s3://BUCKET/versiontest/file9

secondary zone: $ s3cmd ls -r s3://BUCKET
2018-05-31 15:08        41   s3://BUCKET/dummy
2018-05-31 15:08         6   s3://BUCKET/versiontest/file1
2018-05-31 15:08         6   s3://BUCKET/versiontest/file6
2018-05-31 15:08         6   s3://BUCKET/versiontest/file7


Expected results:

Identical bucket listing on both zones.

Additional info:

Comment 26 errata-xmlrpc 2018-07-26 18:06:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2261


Note You need to log in before you can comment on or make changes to this bug.