Description of problem:
In a multisite configuration where PUTs and DELETEs on the same object race to complete on one zone, other zones are unable to resolve that race during data sync.
This happens when DELETE loses this race (getting an ECANCELED error from the osd), because it writes a successful completion entry to the bucket index log instead of a canceled entry.
Fairly reproducible in workloads that PUT and DELETE to the same object. The upstream tracker issue http://tracker.ceph.com/issues/22804 includes a cosbench workload that reproduces
Source and destination zones contain a different set of objects after data sync completes.
Source and destination zones contain the same set of objects after data sync.
*** Bug 1567938 has been marked as a duplicate of this bug. ***