2293611 – [RDR] token-exchange-agent pod Instability after upgrade to ODF 4.16

Bug 2293611 - [RDR] token-exchange-agent pod Instability after upgrade to ODF 4.16

Summary: [RDR] token-exchange-agent pod Instability after upgrade to ODF 4.16

Keywords:
Status:	ASSIGNED
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	odf-dr
Sub Component:
Version:	4.16
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	ODF 4.16.4
Assignee:	umanga
QA Contact:	Sidhant Agrawal
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	2293610 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2024-06-21 08:50 UTC by Sidhant Agrawal
Modified:	2024-10-04 15:16 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	.Instability of the token-exchange-agent pod after upgrade The token-exchange-agent pod on the managed cluster is unstable as the old deployment resources are not cleaned up properly. This might cause application failover action to fail. Workaround: Refer the knowledgebase article, "token-exchange-agent" pod on managed cluster is unstable after upgrade to ODF 4.16.0. Result: If the workaround is followed, "token-exchange-agent" pod is stabilized and failover action works as expected.
Clone Of:
Environment:
Last Closed:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OCSBZM-8551	0	None	None	None	2024-07-16 11:03:06 UTC

Description Sidhant Agrawal 2024-06-21 08:50:17 UTC

Description of problem (please be detailed as possible and provide log
snippests):
On a RDR setup after upgrading the cluster from 4.15.3 to 4.16.0-130, it has been observed that the "token-exchange-agent" pod on managed cluster is unstable. The pod crashes few times, with a new pod spinning up to replace it. Sometimes there is no crash, but still new pod comes up. This cycle continues indefinitely.

> Case 1 (pod logs in Additional Info section):
token-exchange-agent-6d994bf479-7w545                             0/1     CrashLoopBackOff   3 (39s ago)   9m25s
--
token-exchange-agent-6d994bf479-7w545                             0/1     Running             4 (71s ago)   9m57s
token-exchange-agent-76cf9f4f9b-47mcm                             0/1     ContainerCreating   0             1s
--
token-exchange-agent-6d994bf479-7w545                             0/1     Running   4 (75s ago)   10m
token-exchange-agent-76cf9f4f9b-47mcm                             0/1     Running   0             5s
--
token-exchange-agent-76cf9f4f9b-47mcm                             1/1     Running   0          119s


> Case 2:

token-exchange-agent-5c59bb74cb-g2hjs                             1/1     Running   0             11m
token-exchange-agent-5d559f946c-x77jh                             0/1     Running   0             59s
--
token-exchange-agent-5d559f946c-x77jh                             1/1     Running   0             3m26s

Version of all relevant components (if applicable):
ODF: 4.16.0-130.stable
OCP: 4.16.0-0.nightly-2024-06-19-080432
ACM: 2.10.3
Submariner: 0.17.1
Gitops: 1.12.3
Volsync: 0.9.1
OADP: 1.3.2

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?


Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Configure RDR setup using OCP 4.15 and ODF 4.15
2. Perform upgrade
3. Observe the token-exchange-agent pod on managed clusters

Actual results:
Pod crashes and restarts

Expected results:
Pod should be stable and doesn't crash or restart indefinitely


Additional Info:

$ oc get csv -n openshift-storage
NAME                                         DISPLAY                            VERSION             REPLACES                                PHASE
mcg-operator.v4.16.0-130.stable              NooBaa Operator                    4.16.0-130.stable   mcg-operator.v4.15.3-rhodf              Succeeded
ocs-client-operator.v4.16.0-130.stable       OpenShift Data Foundation Client   4.16.0-130.stable                                           Succeeded
ocs-operator.v4.16.0-130.stable              OpenShift Container Storage        4.16.0-130.stable   ocs-operator.v4.15.3-rhodf              Succeeded
odf-csi-addons-operator.v4.16.0-130.stable   CSI Addons                         4.16.0-130.stable   odf-csi-addons-operator.v4.15.3-rhodf   Succeeded
odf-operator.v4.16.0-130.stable              OpenShift Data Foundation          4.16.0-130.stable   odf-operator.v4.15.3-rhodf              Succeeded
odf-prometheus-operator.v4.16.0-130.stable   Prometheus Operator                4.16.0-130.stable                                           Succeeded
odr-cluster-operator.v4.16.0-130.stable      Openshift DR Cluster Operator      4.16.0-130.stable   odr-cluster-operator.v4.15.3-rhodf      Succeeded
openshift-gitops-operator.v1.12.3            Red Hat OpenShift GitOps           1.12.3              openshift-gitops-operator.v1.12.2       Succeeded
recipe.v4.16.0-130.stable                    Recipe                             4.16.0-130.stable                                           Succeeded
rook-ceph-operator.v4.16.0-130.stable        Rook-Ceph                          4.16.0-130.stable                                           Succeeded
volsync-product.v0.9.1                       VolSync                            0.9.1               volsync-product.v0.8.2                  Succeeded


token-exchange-agent Pod logs when there is a crash (case 1):

I0621 08:37:09.389256       1 manager.go:112] Starting addon agents.
I0621 08:37:09.389652       1 common.go:40] Health probes server is running.
I0621 08:37:09.391274       1 manager.go:196] starting hub controller manager
I0621 08:37:09.391347       1 manager.go:309] starting spoke controller manager
I0621 08:37:09.391397       1 manager.go:240] Waiting for MaintenanceMode CRD to be created. MaintenanceMode controller is not running yet.
I0621 08:37:09.391555       1 manager.go:275] Starting lease updater.
I0621 08:37:09.404192       1 manager.go:262] MaintenanceMode CRD exists. MaintenanceMode controller is now running.
W0621 08:37:09.418144       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:37:09.418172       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
W0621 08:37:09.420068       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:37:09.420104       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:37:09.604552       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:09.605690       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:09.810062       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:09.820636       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:09.820735       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:09.832760       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:09.832933       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:09.855429       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:09.855558       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:09.897718       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:09.897832       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:09.915274       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:09.927227       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:09.949149       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:09.980000       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:09.980141       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:09.991036       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:10.073710       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:10.142209       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:10.142389       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:10.236610       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:37:10.340115       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:37:10.340142       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
I0621 08:37:10.465114       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:10.465250       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:10.559231       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:37:10.769452       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:37:10.769483       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:37:11.107172       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:11.107294       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:11.201198       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
I0621 08:37:12.389177       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:12.389363       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:12.483282       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:37:13.540994       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:37:13.541096       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
W0621 08:37:13.562591       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:37:13.562620       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:37:14.952517       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:14.952670       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:15.046223       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:37:18.724804       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:37:18.724835       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
W0621 08:37:19.468938       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:37:19.468987       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:37:20.076072       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:20.076194       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:20.168655       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:37:28.616033       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:37:28.616074       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:37:30.319051       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:30.319208       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:30.411292       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:37:30.572544       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:37:30.572578       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
W0621 08:37:47.003710       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:37:47.003739       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
W0621 08:37:47.713311       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:37:47.713344       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
	>  goroutine 155 [running]:
	>  runtime/debug.Stack()
	>  	/usr/lib/golang/src/runtime/debug/stack.go:24 +0x5e
	>  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/log/log.go:60 +0xcd
	>  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).WithValues(0xc000128200, {0xc000d163c0, 0x4, 0x4})
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/log/deleg.go:168 +0x49
	>  github.com/go-logr/logr.Logger.WithValues(...)
	>  	/remote-source/deps/gomod/pkg/mod/github.com/go-logr/logr.1/logr.go:332
	>  sigs.k8s.io/controller-runtime/pkg/builder.(*Builder).doController.func1(0xc000a839a0)
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/builder/controller.go:402 +0x2ba
	>  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000280f00, {0x27802a0, 0xc0006e4370}, {0x212b280?, 0xc000a83140?})
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/internal/controller/controller.go:306 +0x16a
	>  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000280f00, {0x27802a0, 0xc0006e4370})
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/internal/controller/controller.go:266 +0x1af
	>  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/internal/controller/controller.go:227 +0x79
	>  created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 147
	>  	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/internal/controller/controller.go:223 +0x565
I0621 08:37:50.800948       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:37:50.801066       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:37:50.893760       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:38:26.849200       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:38:26.849235       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
W0621 08:38:30.087863       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:38:30.087895       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
I0621 08:38:31.763595       1 blue_secret_controller.go:59] Reconciling blue secretsecretopenshift-storage/cluster-peer-token-ocs-storagecluster-cephcluster
I0621 08:38:31.763732       1 rook_secret_handler.go:67] detected cluster type: Converged
I0621 08:38:31.856349       1 s3_controller.go:70] Reconciling OBCOBCopenshift-storage/odrbucket-01310c056be2
W0621 08:39:02.893120       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
E0621 08:39:02.893155       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "secrets" in API group "" in the namespace "sagrawal-c1"
W0621 08:39:03.565503       1 reflector.go:539] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:39:03.565532       1 reflector.go:147] pkg/mod/k8s.io/client-go.2/tools/cache/reflector.go:229: Failed to watch *v1alpha1.MirrorPeer: failed to list *v1alpha1.MirrorPeer: mirrorpeers.multicluster.odf.openshift.io is forbidden: User "system:open-cluster-management:cluster:sagrawal-c1:addon:maintenance:agent:maintenance" cannot list resource "mirrorpeers" in API group "multicluster.odf.openshift.io" at the cluster scope
E0621 08:39:09.392659       1 manager.go:198] failed to wait for greensecret_controller caches to sync: timed out waiting for cache to be synced for Kind *v1.Secretproblem running hub controller manager

Comment 4 Sunil Kumar Acharya 2024-06-24 06:42:03 UTC

*** Bug 2293610 has been marked as a duplicate of this bug. ***

Comment 5 Sunil Kumar Acharya 2024-06-24 08:09:49 UTC

Moving the non-blocker BZ out of ODF-4.16.0. If this is a blocker, feel free to propose it back with justification note.

Note You need to log in before you can comment on or make changes to this bug.