Description of problem (please be detailed as possible and provide log
snippests):
Observing noobaa restarts multiple times on RDR Longevity setup.

Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Keep the RDR cluster with replication going on for a longer duration(in this case 2 months)
2. Noobaa operators restarts multiple times (86 times on c1, 75 times on c2)

Log message in noobaa operator pod

time="2023-08-02T07:03:09Z" level=info msg="❌ Not Found:  \"noobaa-default-backing-store-noobaa-noobaa\"\n"

oc get backingstore -n openshift-storage
NAME                           TYPE            PHASE   AGE
noobaa-default-backing-store   s3-compatible   Ready   55d


Actual results:
Too many restarts are not expected

Expected results:

Additional info:
1) No RDR operations were performed.

Must-gather logs:-

c1 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/Longevity/ceph-mon-restart/c1/

c2 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/Longevity/ceph-mon-restart/c2/

hub - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/Longevity/ceph-mon-restart/hub/

Live cluster is avaiable for debugging

hub - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/25311/

c1 - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/25313/

c2 - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/25312/


Version details

ODF- 4.13.0-219
OCP - 4.13.0-0.nightly-2023-06-05-164816
ACM - 2.8
Submariner - 0.15.1
MCO - 4.13.0-219
ceph - ceph version 17.2.6-70.0.TEST.bz2119217.el9cp (6d74fefa15d1216867d1d112b47bb83c4913d28f) quincy (stable)