Description of problem:
Create 100 buckets and upload object in each bucket on both rgw zones in parallel. The objects sync successfully to master zone whereas on the non-master zone, the buckets are synced from master but not the objects
Version-Release number of selected component (if applicable):
ceph-radosgw-10.2.2-5.el7cp.x86_64
How reproducible:
Always
Steps to Reproduce:
1. Start a boto script on each zone to create and upload 100 buckets with an object of 1.5G in each bucket.
2. All objects and buckets are synced to master but only buckets are synced to non-master zone. No objects are created.
Actual results:
On master:
]# radosgw-admin sync status --rgw-zone=us-1 --debug-rgw=0 --debug-ms=0 realm 1c60c863-689d-441f-b370-62390562e2aa (earth)
zonegroup 540c9b3f-5eb7-4a67-a581-54bc704ce827 (us)
zone 505a3a8e-19cf-4295-a43d-559e763891f6 (us-1)
metadata sync no sync (zone is master)
data sync source: d48cb942-a5fa-4597-89fd-0bab3bb9c5a3 (us-2)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
On non-master:
# radosgw-admin sync status --rgw-zone=us-2 --debug-rgw=0 --debug-ms=0
realm 1c60c863-689d-441f-b370-62390562e2aa (earth)
zonegroup 540c9b3f-5eb7-4a67-a581-54bc704ce827 (us)
zone d48cb942-a5fa-4597-89fd-0bab3bb9c5a3 (us-2)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is behind on 1 shards
oldest incremental change not applied: 2016-06-23 09:57:35.0.097857s
data sync source: 505a3a8e-19cf-4295-a43d-559e763891f6 (us-1)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 8 shards
oldest incremental change not applied: 2016-06-29 07:34:15.0.194232s
Expected results:
Sync should work from both side simultaneously
Additional info:
Will provide the path to logs
It seems that data sync stopped following errors in requests from non-master to master (retrieving of all incremental data log shards failed, maybe timing out due to lack of request processing threads on the master).
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2016-1755.html
Description of problem: Create 100 buckets and upload object in each bucket on both rgw zones in parallel. The objects sync successfully to master zone whereas on the non-master zone, the buckets are synced from master but not the objects Version-Release number of selected component (if applicable): ceph-radosgw-10.2.2-5.el7cp.x86_64 How reproducible: Always Steps to Reproduce: 1. Start a boto script on each zone to create and upload 100 buckets with an object of 1.5G in each bucket. 2. All objects and buckets are synced to master but only buckets are synced to non-master zone. No objects are created. Actual results: On master: ]# radosgw-admin sync status --rgw-zone=us-1 --debug-rgw=0 --debug-ms=0 realm 1c60c863-689d-441f-b370-62390562e2aa (earth) zonegroup 540c9b3f-5eb7-4a67-a581-54bc704ce827 (us) zone 505a3a8e-19cf-4295-a43d-559e763891f6 (us-1) metadata sync no sync (zone is master) data sync source: d48cb942-a5fa-4597-89fd-0bab3bb9c5a3 (us-2) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source On non-master: # radosgw-admin sync status --rgw-zone=us-2 --debug-rgw=0 --debug-ms=0 realm 1c60c863-689d-441f-b370-62390562e2aa (earth) zonegroup 540c9b3f-5eb7-4a67-a581-54bc704ce827 (us) zone d48cb942-a5fa-4597-89fd-0bab3bb9c5a3 (us-2) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is behind on 1 shards oldest incremental change not applied: 2016-06-23 09:57:35.0.097857s data sync source: 505a3a8e-19cf-4295-a43d-559e763891f6 (us-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 8 shards oldest incremental change not applied: 2016-06-29 07:34:15.0.194232s Expected results: Sync should work from both side simultaneously Additional info: Will provide the path to logs