Description of problem:

's3cmd sync' has some logic that compares the md5sum of local objects, and uses COPY for matching objects rather than uploading them separately with PUT. After COPYing objects, it then sets ACLs on the copied objects as a separate operation.

This second PutACL operation is treated as a separate transaction on the bucket index with a newer timestamp, so multisite sync chooses this operation as the most recent version to apply. However, this bucket index operation does not carry all of the necessary information related to object versioning - this information is only present in the 'link_olh' operation, which is only generated by the initial PUT or COPY.

As a result, when sync tries to process this PutACL operation, it skips the object with a message like:

> bucket skipping sync obj: versioned object will be synced on link_olh


Version-Release number of selected component (if applicable):

All versions of RHCS.


How reproducible:

Fairly reproduceable for objects uploaded with COPY. 


Steps to Reproduce:

1. Deploy a basic multisite configuration with two zones.
2. Create a BUCKET on the primary zone.
3. Enable versioning on the BUCKET.
4. Upload a single dummy object to the BUCKET. (this will trigger a full sync so that later objects are processed during incremental sync)
5. Create a local directory with 10 identical files:

$ mkdir versiontest
$ for i in `seq 0 9` ; do echo hello > versiontest/file$i ; done

6. Sync this directory into the bucket:

$ s3cmd sync versiontest s3://BUCKET

7. Let the bucket sync, then list the bucket on both zones.


Actual results:

Some objects are not present on the secondary zone.

primary zone: $ s3cmd ls -r s3://BUCKET
2018-05-31 15:08        41   s3://BUCKET/dummy
2018-05-31 15:08         6   s3://BUCKET/versiontest/file1
2018-05-31 15:08         6   s3://BUCKET/versiontest/file10
2018-05-31 15:08         6   s3://BUCKET/versiontest/file2
2018-05-31 15:08         6   s3://BUCKET/versiontest/file3
2018-05-31 15:08         6   s3://BUCKET/versiontest/file4
2018-05-31 15:08         6   s3://BUCKET/versiontest/file5
2018-05-31 15:08         6   s3://BUCKET/versiontest/file6
2018-05-31 15:08         6   s3://BUCKET/versiontest/file7
2018-05-31 15:08         6   s3://BUCKET/versiontest/file8
2018-05-31 15:08         6   s3://BUCKET/versiontest/file9

secondary zone: $ s3cmd ls -r s3://BUCKET
2018-05-31 15:08        41   s3://BUCKET/dummy
2018-05-31 15:08         6   s3://BUCKET/versiontest/file1
2018-05-31 15:08         6   s3://BUCKET/versiontest/file6
2018-05-31 15:08         6   s3://BUCKET/versiontest/file7


Expected results:

Identical bucket listing on both zones.

Additional info: