Bug 2274750

Summary:	[RDR] [Tracker ACM-10967] Pods in NS openshift-operators on hub with ACM observability enabled starts crashing after a few days
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Aman Agrawal <amagrawa>
Component:	odf-dr	Assignee:	Karolin Seeger <kseeger>
odf-dr sub component:	ramen	QA Contact:	Aman Agrawal <amagrawa>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	unspecified	CC:	bmekhiss, kseeger, muagarwa, sheggodu
Version:	4.15	Keywords:	Regression
Target Milestone:	---
Target Release:	ODF 4.16.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.16.0-89	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-07-17 13:19:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Aman Agrawal 2024-04-12 17:00:30 UTC

Description of problem (please be detailed as possible and provide log
snippests):


Version of all relevant components (if applicable):
OCP 4.15.0-0.nightly-2024-04-02-181803
ACM 2.10 GA'ed (ACM wasn't upgraded, it's a fresh install)
ODF 4.15.1-1
ceph version 17.2.6-196.el9cp (cbbf2cfb549196ca18c0c9caff9124d83ed681a4) quincy (stable)
MCE 2.5.1
OADP 1.3.0
Submariner 0.17.0
VolSync 0.9.0

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Create a RDR setup with ACM observability enabled and deploy a few DR protected workloads.
2. Monitor the pods on hub in NS openshift-operators and for errors on the DR dashboard on the RHACM console.
3.


Actual results:

On hub:

amanagrawal@Amans-MacBook-Pro ~ % oc get pods -n openshift-operators
NAME                                       READY   STATUS    RESTARTS      AGE
odfmo-controller-manager-55b5f5c56b-st4bx   0/1     CrashLoopBackOff   132 (105s ago)   8d
ramen-hub-operator-669f85b468-7j6zs         1/2     CrashLoopBackOff   376 (38s ago)    8d

We see these 2 pods are crashing and can't recover

Disaster recovery health under Operator status of DR dashboard was degraded.


Additionally, @bmekhiss helped us discover that 4926 secrets are under open-cluster-management-observability NS on hub which is the root cause underlying in ACM observability.

Issue discussed here with ACM team- https://redhat-internal.slack.com/archives/CUU609ZQC/p1712928814274489


Expected results: Pods in NS openshift-operators shouldn't crash on a RDR setup with ACM observability enabled on it


Additional info:

Comment 13 errata-xmlrpc 2024-07-17 13:19:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591