Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1349355

Summary: Multisite: Unable to sync objects created by tenant users
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: shilpa <smanjara>
Component: RGWAssignee: Casey Bodley <cbodley>
Status: CLOSED ERRATA QA Contact: shilpa <smanjara>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: cbodley, ceph-eng-bugs, ceph-qe-bugs, hnallurv, kbader, kdreyer, mbenjamin, owasserm, sweil
Target Milestone: rc   
Target Release: 2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.2-20.el7cp Ubuntu: ceph_10.2.2-15redhat1xenial Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-23 19:42:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
master zone logs
none
magna075 logs none

Description shilpa 2016-06-23 10:04:46 UTC 
Description of problem:
Objects created by tenant users are not synced across zones.


Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.2-5.el7cp.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Configure multisite between two clusters with one rgw node in each.
2. Create a tenant user on master zone and create bucket with objects 
3. Verify if the buckets and objects are synced to the peer zone

Actual results:
Buckets sync but objects fail to sync with errors in logs:

7fc636ffd700  0 ERROR: failed to get bucket instance info for bucket id=test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10
7fc5997fa700 20 sending request to http://magna075:8080/admin/log/?type=data&id=12&info&rgwx-zonegroup=540c9b3f-5eb7-4a67-a581-54bc704ce827
7fc62f7fe700  0 rgw meta sync: ERROR: can't remove key: bucket.instance:test5:505a3a8e-19cf-4295-a43d-559e763891f6.4176.10 ret=-2
7fc5997fa700  0 ERROR: failed to retrieve bucket info for bucket=test5 bucket_id=505a3a8e-19cf-4295-a43d-559e763891f6.4176.10


Logs with debug are uploaded. The test buckets are 'test3', 'test5' and 'test6' all created on master zone magna115.
Comment 2 shilpa 2016-06-23 10:10:37 UTC 
Created attachment 1171398 [details]
master zone logs
Comment 3 shilpa 2016-06-23 10:11:23 UTC 
Created attachment 1171399 [details]
magna075 logs
Comment 5 Casey Bodley 2016-06-24 16:01:07 UTC 
More info from Shilpa on creating and using tenant users:

radosgw-admin user create --uid=testuser --tenant=test --display-name="Test User"
radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --tenant=test --access=full
radosgw-admin key create --subuser=testuser:swift --uid=testuser  --tenant=test --key-type=swift --gen-secret
swift -A http://magna115:8080/auth/1.0 -U 'test$testuser:swift' -K '<secret>' list
Comment 6 Casey Bodley 2016-06-24 16:06:50 UTC 
What I've learned so far:

Each tenant has a separate 'namespace' for buckets, which works by adding the tenant name to the beginning of the bucket/instance object names.

The multisite sync process doesn't track tenants, so it's looking for objects without this tenant prefix and failing to find and sync them.
Comment 8 Casey Bodley 2016-07-06 21:00:31 UTC 
added upstream pull request at https://github.com/ceph/ceph/pull/10075. pending further testing and review
Comment 18 shilpa 2016-07-20 09:17:44 UTC 
Verified on ceph-10.2.2-23
Comment 20 errata-xmlrpc 2016-08-23 19:42:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1755.html