Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2228612

Summary:	[5.3][RGW][archive]: recovering shards in sync status on archive site upon a bucket delete from primary
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	shilpa <smanjara>
Component:	RGW-Multisite	Assignee:	shilpa <smanjara>
Status:	CLOSED ERRATA	QA Contact:	Vidushi Mishra <vimishra>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	5.3	CC:	akraj, ceph-eng-bugs, cephqe-warriors, ckulal, mkasturi, tserlin, vereddy, vimishra
Target Milestone:	---
Target Release:	5.3z5
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-16.2.10-195.el8cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-28 09:40:57 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description shilpa 2023-08-02 20:30:13 UTC

This bug was initially created as a copy of Bug #2128421


Description of problem:
Recovering shards seen in 'radosgw-admin sync status' on archive site, when a bucket from primary is deleted.
snippet:
[root@ceph-arc-kvm-5-3-archive-pn959k-node5 cephuser]# radosgw-admin sync status
          realm 2512e36d-cb3f-4955-87ce-a5744fe6a135 (india)
      zonegroup eb4a0d90-aef9-48d6-947e-316e68d414ca (shared)
           zone 363592b3-6183-43c2-ad32-3f80038350bb (archive)
   current time 2022-09-20T13:03:34Z
zonegroup features enabled: resharding
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: a793ad9b-3e3c-48e4-b120-182825478817 (primary)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        1 shards are recovering
                        recovering shards: [124]

~8hours after bucket delete is deleted, 1 recovering shards still exists

Version-Release number of selected component (if applicable):
ceph version 16.2.10-43.el8cp

How reproducible:
2/2

Steps to Reproduce:
1. Ceph cluster deployed with archive zone [ 1 rgw daemon in both singlesite + archive zone]
2. create a bucket on primary and add an object (size2G)
3. allow the sync to complete
4. delete the bucket from primary using 
'radosgw-admin bucket rm  --bucket kvm-dlo-bkt1 --purge-objects'
5. bucket and object is deleted from both primary and archive site
snippet:
primary site:
(env) [root@ceph-pri-kvm-5-3-archive-pn959k-node5 lib]# radosgw-admin bucket stats --bucket kvm-dlo-bkt1
failure: (2002) Unknown error 2002: 

secondary site:
[root@ceph-arc-kvm-5-3-archive-pn959k-node5 ~]# radosgw-admin bucket stats --bucket kvm-dlo-bkt1
failure: (2002) Unknown error 2002: 

6. check sync status on both sites
primary site:
(env) [root@ceph-pri-kvm-5-3-archive-pn959k-node5 lib]# radosgw-admin sync status
          realm 2512e36d-cb3f-4955-87ce-a5744fe6a135 (india)
      zonegroup eb4a0d90-aef9-48d6-947e-316e68d414ca (shared)
           zone a793ad9b-3e3c-48e4-b120-182825478817 (primary)
   current time 2022-09-20T03:51:28Z
zonegroup features enabled: resharding
  metadata sync no sync (zone is master)
      data sync source: 363592b3-6183-43c2-ad32-3f80038350bb (archive)
                        not syncing from zone

secondary site:
after ~8hours the status still has 1 recovering shards 
[root@ceph-arc-kvm-5-3-archive-pn959k-node5 cephuser]# radosgw-admin sync status
          realm 2512e36d-cb3f-4955-87ce-a5744fe6a135 (india)
      zonegroup eb4a0d90-aef9-48d6-947e-316e68d414ca (shared)
           zone 363592b3-6183-43c2-ad32-3f80038350bb (archive)
   current time 2022-09-20T13:03:34Z
zonegroup features enabled: resharding
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: a793ad9b-3e3c-48e4-b120-182825478817 (primary)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        1 shards are recovering
                        recovering shards: [124]
Almost 8 hours since the bucket is deleted, the sync status still holds recovering shards


Actual results:
recovering shards reported in radosgw-admin sync status 

Expected results:
sync status reports appropriate status

Additional info:

setup details: root/r
site1: rgw - 10.0.211.224
archive site:  rgw - 10.0.208.162

PFA, the console logs and rgw logs at http://magna002.ceph.redhat.com/ceph-qe-logs/madhavi/5.3/archive/

Comment 1 RHEL Program Management 2023-08-02 20:30:22 UTC

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 9 errata-xmlrpc 2023-08-28 09:40:57 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.3 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:4760

Comment 10 Red Hat Bugzilla 2023-12-27 04:25:20 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days