Bug 1349355 - Multisite: Unable to sync objects created by tenant users
Summary: Multisite: Unable to sync objects created by tenant users
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 2.0
Assignee: Casey Bodley
QA Contact: shilpa
Depends On:
TreeView+ depends on / blocked
Reported: 2016-06-23 10:04 UTC by shilpa
Modified: 2017-07-30 15:52 UTC (History)
9 users (show)

Fixed In Version: RHEL: ceph-10.2.2-20.el7cp Ubuntu: ceph_10.2.2-15redhat1xenial
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-08-23 19:42:27 UTC
Target Upstream Version:

Attachments (Terms of Use)
master zone logs (3.07 MB, application/zip)
2016-06-23 10:10 UTC, shilpa
no flags Details
magna075 logs (2.99 MB, application/zip)
2016-06-23 10:11 UTC, shilpa
no flags Details

System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 16469 0 None None None 2016-06-24 13:39:08 UTC
Red Hat Product Errata RHBA-2016:1755 0 normal SHIPPED_LIVE Red Hat Ceph Storage 2.0 bug fix and enhancement update 2016-08-23 23:23:52 UTC

Description shilpa 2016-06-23 10:04:46 UTC
Description of problem:
Objects created by tenant users are not synced across zones.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Configure multisite between two clusters with one rgw node in each.
2. Create a tenant user on master zone and create bucket with objects 
3. Verify if the buckets and objects are synced to the peer zone

Actual results:
Buckets sync but objects fail to sync with errors in logs:

7fc636ffd700  0 ERROR: failed to get bucket instance info for bucket id=test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10
7fc5997fa700 20 sending request to http://magna075:8080/admin/log/?type=data&id=12&info&rgwx-zonegroup=540c9b3f-5eb7-4a67-a581-54bc704ce827
7fc62f7fe700  0 rgw meta sync: ERROR: can't remove key: bucket.instance:test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10 ret=-2
7fc5997fa700  0 ERROR: failed to retrieve bucket info for bucket=test5 bucket_id=505a3a8e-19cf-4295-a43d-559e763891f6.4176.10

Logs with debug are uploaded. The test buckets are 'test3', 'test5' and 'test6' all created on master zone magna115.

Comment 2 shilpa 2016-06-23 10:10:37 UTC
Created attachment 1171398 [details]
master zone logs

Comment 3 shilpa 2016-06-23 10:11:23 UTC
Created attachment 1171399 [details]
magna075 logs

Comment 5 Casey Bodley 2016-06-24 16:01:07 UTC
More info from Shilpa on creating and using tenant users:

radosgw-admin user create --uid=testuser --tenant=test --display-name="Test User"
radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --tenant=test --access=full
radosgw-admin key create --subuser=testuser:swift --uid=testuser  --tenant=test --key-type=swift --gen-secret
swift -A http://magna115:8080/auth/1.0 -U 'test$testuser:swift' -K '<secret>' list

Comment 6 Casey Bodley 2016-06-24 16:06:50 UTC
What I've learned so far:

Each tenant has a separate 'namespace' for buckets, which works by adding the tenant name to the beginning of the bucket/instance object names.

The multisite sync process doesn't track tenants, so it's looking for objects without this tenant prefix and failing to find and sync them.

Comment 8 Casey Bodley 2016-07-06 21:00:31 UTC
added upstream pull request at https://github.com/ceph/ceph/pull/10075. pending further testing and review

Comment 18 shilpa 2016-07-20 09:17:44 UTC
Verified on ceph-10.2.2-23

Comment 20 errata-xmlrpc 2016-08-23 19:42:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.