Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use Jira Cloud for all bug tracking management.

Bug 1349355

Summary: Multisite: Unable to sync objects created by tenant users
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: shilpa <smanjara>
Component: RGWAssignee: Casey Bodley <cbodley>
Status: CLOSED ERRATA QA Contact: shilpa <smanjara>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: cbodley, ceph-eng-bugs, ceph-qe-bugs, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.2-20.el7cp Ubuntu: ceph_10.2.2-15redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:42:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
master zone logs
none
magna075 logs none

Description shilpa 2016-06-23 10:04:46 UTC 
Description of problem:
Objects created by tenant users are not synced across zones.


Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.2-5.el7cp.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Configure multisite between two clusters with one rgw node in each.
2. Create a tenant user on master zone and create bucket with objects 
3. Verify if the buckets and objects are synced to the peer zone

Actual results:
Buckets sync but objects fail to sync with errors in logs:

7fc636ffd700  0 ERROR: failed to get bucket instance info for bucket id=test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10
7fc5997fa700 20 sending request to http://magna075:8080/admin/log/?type=data&id=12&info&rgwx-zonegroup=540c9b3f-5eb7-4a67-a581-54bc704ce827
7fc62f7fe700  0 rgw meta sync: ERROR: can't remove key: bucket.instance:test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10 ret=-2
7fc5997fa700  0 ERROR: failed to retrieve bucket info for bucket=test5 bucket_id=505a3a8e-19cf-4295-a43d-559e763891f6.4176.10


Logs with debug are uploaded. The test buckets are 'test3', 'test5' and 'test6' all created on master zone magna115.
Comment 2 shilpa 2016-06-23 10:10:37 UTC 
Created attachment 1171398 [details]
master zone logs
Comment 3 shilpa 2016-06-23 10:11:23 UTC 
Created attachment 1171399 [details]
magna075 logs
Comment 5 Casey Bodley 2016-06-24 16:01:07 UTC 
More info from Shilpa on creating and using tenant users:

radosgw-admin user create --uid=testuser --tenant=test --display-name="Test User"
radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --tenant=test --access=full
radosgw-admin key create --subuser=testuser:swift --uid=testuser  --tenant=test --key-type=swift --gen-secret
swift -A http://magna115:8080/auth/1.0 -U 'test$testuser:swift' -K '<secret>' list
Comment 6 Casey Bodley 2016-06-24 16:06:50 UTC 
What I've learned so far:

Each tenant has a separate 'namespace' for buckets, which works by adding the tenant name to the beginning of the bucket/instance object names.

The multisite sync process doesn't track tenants, so it's looking for objects without this tenant prefix and failing to find and sync them.
Comment 8 Casey Bodley 2016-07-06 21:00:31 UTC 
added upstream pull request at https://github.com/ceph/ceph/pull/10075. pending further testing and review
Comment 18 shilpa 2016-07-20 09:17:44 UTC 
Verified on ceph-10.2.2-23
Comment 20 errata-xmlrpc 2016-08-23 19:42:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html