Bug 1584763
| Summary: | Some versioned objects don't sync when uploaded with 's3cmd sync' | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Casey Bodley <cbodley> | |
| Component: | RGW-Multisite | Assignee: | Casey Bodley <cbodley> | |
| Status: | CLOSED ERRATA | QA Contact: | Tejas <tchandra> | |
| Severity: | high | Docs Contact: | Aron Gunn <agunn> | |
| Priority: | high | |||
| Version: | 2.5 | CC: | agunn, cbodley, ceph-eng-bugs, ceph-qe-bugs, tchandra, tserlin, vumrao | |
| Target Milestone: | z1 | |||
| Target Release: | 2.5 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | RHEL: ceph-10.2.10-27.el7cp Ubuntu: ceph_10.2.10-22redhat1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1585238 1585239 (view as bug list) | Environment: | ||
| Last Closed: | 2018-07-26 18:06:48 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1585238, 1585239 | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2261 |
Description of problem: 's3cmd sync' has some logic that compares the md5sum of local objects, and uses COPY for matching objects rather than uploading them separately with PUT. After COPYing objects, it then sets ACLs on the copied objects as a separate operation. This second PutACL operation is treated as a separate transaction on the bucket index with a newer timestamp, so multisite sync chooses this operation as the most recent version to apply. However, this bucket index operation does not carry all of the necessary information related to object versioning - this information is only present in the 'link_olh' operation, which is only generated by the initial PUT or COPY. As a result, when sync tries to process this PutACL operation, it skips the object with a message like: > bucket skipping sync obj: versioned object will be synced on link_olh Version-Release number of selected component (if applicable): All versions of RHCS. How reproducible: Fairly reproduceable for objects uploaded with COPY. Steps to Reproduce: 1. Deploy a basic multisite configuration with two zones. 2. Create a BUCKET on the primary zone. 3. Enable versioning on the BUCKET. 4. Upload a single dummy object to the BUCKET. (this will trigger a full sync so that later objects are processed during incremental sync) 5. Create a local directory with 10 identical files: $ mkdir versiontest $ for i in `seq 0 9` ; do echo hello > versiontest/file$i ; done 6. Sync this directory into the bucket: $ s3cmd sync versiontest s3://BUCKET 7. Let the bucket sync, then list the bucket on both zones. Actual results: Some objects are not present on the secondary zone. primary zone: $ s3cmd ls -r s3://BUCKET 2018-05-31 15:08 41 s3://BUCKET/dummy 2018-05-31 15:08 6 s3://BUCKET/versiontest/file1 2018-05-31 15:08 6 s3://BUCKET/versiontest/file10 2018-05-31 15:08 6 s3://BUCKET/versiontest/file2 2018-05-31 15:08 6 s3://BUCKET/versiontest/file3 2018-05-31 15:08 6 s3://BUCKET/versiontest/file4 2018-05-31 15:08 6 s3://BUCKET/versiontest/file5 2018-05-31 15:08 6 s3://BUCKET/versiontest/file6 2018-05-31 15:08 6 s3://BUCKET/versiontest/file7 2018-05-31 15:08 6 s3://BUCKET/versiontest/file8 2018-05-31 15:08 6 s3://BUCKET/versiontest/file9 secondary zone: $ s3cmd ls -r s3://BUCKET 2018-05-31 15:08 41 s3://BUCKET/dummy 2018-05-31 15:08 6 s3://BUCKET/versiontest/file1 2018-05-31 15:08 6 s3://BUCKET/versiontest/file6 2018-05-31 15:08 6 s3://BUCKET/versiontest/file7 Expected results: Identical bucket listing on both zones. Additional info: