Bug 2229863

Summary:	[RDR] Noobaa operator restarts multiple times on RDR longevity setup
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	kmanohar
Component:	Multi-Cloud Object Gateway	Assignee:	Nimrod Becker <nbecker>
Status:	CLOSED DUPLICATE	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.13	CC:	odf-bz-bot
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-09 08:56:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description kmanohar 2023-08-08 04:54:06 UTC

Description of problem (please be detailed as possible and provide log
snippests):
Observing noobaa restarts multiple times on RDR Longevity setup.

Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Keep the RDR cluster with replication going on for a longer duration(in this case 2 months)
2. Noobaa operators restarts multiple times (86 times on c1, 75 times on c2)

Log message in noobaa operator pod

time="2023-08-02T07:03:09Z" level=info msg="❌ Not Found:  \"noobaa-default-backing-store-noobaa-noobaa\"\n"

oc get backingstore -n openshift-storage
NAME                           TYPE            PHASE   AGE
noobaa-default-backing-store   s3-compatible   Ready   55d


Actual results:
Too many restarts are not expected

Expected results:

Additional info:
1) No RDR operations were performed.

Must-gather logs:-

c1 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/Longevity/ceph-mon-restart/c1/

c2 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/Longevity/ceph-mon-restart/c2/

hub - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/Longevity/ceph-mon-restart/hub/

Live cluster is avaiable for debugging

hub - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/25311/

c1 - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/25313/

c2 - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/25312/


Version details

ODF- 4.13.0-219
OCP - 4.13.0-0.nightly-2023-06-05-164816
ACM - 2.8
Submariner - 0.15.1
MCO - 4.13.0-219
ceph - ceph version 17.2.6-70.0.TEST.bz2119217.el9cp (6d74fefa15d1216867d1d112b47bb83c4913d28f) quincy (stable)

Comment 2 Nimrod Becker 2023-08-09 08:56:28 UTC

Please verify with 4.13.1, this issue is the same as https://bugzilla.redhat.com/show_bug.cgi?id=2216401

*** This bug has been marked as a duplicate of bug 2216401 ***