Description of problem:
RGW multisite stuck on a shard
After the upgrade to RHCS 3.1 we're still facing some replication issues. After the upgrade "radosgw-admin sync status" reported a few shards that are recovering and new objects were created in the dc_zone.rgw.buckets.data pool. But then it got stuck on shard 64 and is still stuck on that shard:
realm <removed name>
zonegroup <removed name>
zone <removed name>
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: <removed name>
full sync: 0/128 shards
incremental sync: 128/128 shards
1 shards are recovering
recovering shards: 
Version-Release number of selected component (if applicable):
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.