Bug 2131932

Summary:	[RGW-MS] RGW multisite sync is stuck after 4.3z1 to 5.3 upgrade
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Ameena Suhani S H <amsyedha>
Component:	RGW-Multisite	Assignee:	Matt Benjamin (redhat) <mbenjamin>
Status:	CLOSED ERRATA	QA Contact:	Hemanth Sai <hmaheswa>
Severity:	urgent	Docs Contact:	Akash Raj <akraj>
Priority:	unspecified
Version:	5.3	CC:	akraj, ceph-eng-bugs, cephqe-warriors, mbenjamin, mkasturi, racpatel, tserlin, vereddy, vimishra
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	5.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-16.2.10-92.el8cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-11 17:41:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2126049

Description Ameena Suhani S H 2022-10-04 07:12:06 UTC

Description of problem:
RGW multisite sync is stuck after 4.3z1 to 5.3 upgrade

Version-Release number of selected component (if applicable):
16.2.10-48.el8cp

How reproducible:
1/1

Steps to Reproduce:
install 4.3z1
Fill - 7- 10% cluster full with fill
Measure - 1hr hybrid 
Upgrade - 10hr hybrid with Upgrade - 10hr (Cluster upgrade to 5.3 in parallel)
Measure post upgrade- 1hr hybrid - 1hr hybrid before enabling resharding
Measure with resharding - Enable resharding and start 1hr hybrid immediately after enabling 
Aging - 2hr hybrid - 
Measure - 1hr hybrid - 1hr
48-72 hrs no I/O


Actual results:
Data is not synced after  around 12 hours of completing the last run

Expected results:
Data should be synced as soon as the workload completed

Additional info:

COSBench URL:
Site1: http://dell-r630-002.dsal.lab.eng.rdu2.redhat.com:19088/controller/index.html
Site2: http://dell-r640-072.dsal.lab.eng.tlv2.redhat.com:19088/controller/index.html

eg: We see that these buckets "bucketpri1" and "bucketsec1" do not have consistent objects across sites

Comment 30 errata-xmlrpc 2023-01-11 17:41:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 5.3 security update and Bug Fix), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0076

Comment 31 Red Hat Bugzilla 2023-09-19 04:27:40 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days