Description of problem: Objects created by tenant users are not synced across zones. Version-Release number of selected component (if applicable): ceph-radosgw-10.2.2-5.el7cp.x86_64 How reproducible: Always Steps to Reproduce: 1. Configure multisite between two clusters with one rgw node in each. 2. Create a tenant user on master zone and create bucket with objects 3. Verify if the buckets and objects are synced to the peer zone Actual results: Buckets sync but objects fail to sync with errors in logs: 7fc636ffd700 0 ERROR: failed to get bucket instance info for bucket id=test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10 7fc5997fa700 20 sending request to http://magna075:8080/admin/log/?type=data&id=12&info&rgwx-zonegroup=540c9b3f-5eb7-4a67-a581-54bc704ce827 7fc62f7fe700 0 rgw meta sync: ERROR: can't remove key: bucket.instance:test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10 ret=-2 7fc5997fa700 0 ERROR: failed to retrieve bucket info for bucket=test5 bucket_id=505a3a8e-19cf-4295-a43d-559e763891f6.4176.10 Logs with debug are uploaded. The test buckets are 'test3', 'test5' and 'test6' all created on master zone magna115.
Created attachment 1171398 [details] master zone logs
Created attachment 1171399 [details] magna075 logs
More info from Shilpa on creating and using tenant users: radosgw-admin user create --uid=testuser --tenant=test --display-name="Test User" radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --tenant=test --access=full radosgw-admin key create --subuser=testuser:swift --uid=testuser --tenant=test --key-type=swift --gen-secret swift -A http://magna115:8080/auth/1.0 -U 'test$testuser:swift' -K '<secret>' list
What I've learned so far: Each tenant has a separate 'namespace' for buckets, which works by adding the tenant name to the beginning of the bucket/instance object names. The multisite sync process doesn't track tenants, so it's looking for objects without this tenant prefix and failing to find and sync them.
added upstream pull request at https://github.com/ceph/ceph/pull/10075. pending further testing and review
Verified on ceph-10.2.2-23
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1755.html