Bug 1992445 - ms - sync status stuck at '83 shards are recovering' in primary cluster when upgraded to rhcs 5.0
Summary: ms - sync status stuck at '83 shards are recovering' in primary cluster when ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW-Multisite
Version: 5.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 5.0
Assignee: Casey Bodley
QA Contact: Vasishta
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-11 06:32 UTC by Vasishta
Modified: 2021-08-30 08:32 UTC (History)
5 users (show)

Fixed In Version: ceph-16.2.0-115.el8cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-08-30 08:31:54 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 52128 0 None None None 2021-08-11 12:49:33 UTC
Github ceph ceph pull 42740 0 None None None 2021-08-11 12:49:33 UTC
Red Hat Issue Tracker RHCEPH-697 0 None None None 2021-08-30 00:23:55 UTC
Red Hat Product Errata RHBA-2021:3294 0 None None None 2021-08-30 08:32:05 UTC

Description Vasishta 2021-08-11 06:32:31 UTC
Description of problem: and Steps to Reproduce:
1) Configured two RHCS 4.x clusters with single RGW daemon per cluster
with RGW MS configuration using ceph-ansible
Both the cluster were baremetal clusters
2) Migrated primary cluster to containerized cluster
Upgraded primary cluster to RHCS 5.0
cephadm-adopt was initiated on primary cluster
3) During first two steps rgw bucket/object (using boto scripts) /user creation and deletion were tried on primary cluster

4) After cephadm-adopt was completed on primary cluster, it was observed that sync status was stuck saying '83 shards are recovering' Even after 13+ hrs of observation, where as 
Secondary site always mentioned data is caught up with source                


Version-Release number of selected component (if applicable):
16.2.0-114.el8cp

How reproducible:
Not yet tried to reproduced
This issue was hit when tried to reproduce Bug 1989849

Actual results:
data sync source: 62995596-f766-4cd3-8399-c85d9d8998e3 (US_WEST)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        83 shards are recovering
                        recovering shards: [0,1,2,8,9,10,11,12,13,14,15,16,17,18,24,25,26,27,28,29,30,31,32,33,34,40,41,42,43,44,45,46,47,48,49,50,56,57,58,59,60,61,62,63,64,65,66,72,74,75,76,77,78,79,80,88,89,90,91,92,93,94,95,96,97,98,104,105,106,107,108,109,110,111,112,113,114,120,121,122,123,124,127]


Expected results:
sync status must get updated accordingly

Additional info:
Secondary site -
data sync source: 6a12b35e-caf9-4cac-aef6-e0d98335537a (US_EAST)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

Please refer -
- https://bugzilla.redhat.com/show_bug.cgi?id=1989849#c8
- https://bugzilla.redhat.com/show_bug.cgi?id=1989849#c9

Comment 1 RHEL Program Management 2021-08-11 06:32:38 UTC
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 10 errata-xmlrpc 2021-08-30 08:31:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294


Note You need to log in before you can comment on or make changes to this bug.