Bug 2227017

Summary: [RDR] token-exchange-agent pod in CrashLoopBackOff state
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Sidhant Agrawal <sagrawal>
Component: odf-drAssignee: umanga <uchapaga>
odf-dr sub component: multicluster-orchestrator QA Contact: Sidhant Agrawal <sagrawal>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: muagarwa, odf-bz-bot
Version: 4.14Keywords: Regression, TestBlocker
Target Milestone: ---   
Target Release: ODF 4.14.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.14.0-93 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2232555 (view as bug list) Environment:
Last Closed: 2023-11-08 18:52:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2232555    

Description Sidhant Agrawal 2023-07-27 13:00:09 UTC 
Description of problem (please be detailed as possible and provide log
snippests):
using OCP 4.14, while configuring RDR setup, after creating DRPolicy on hub cluster, observed that token-exchange-agent pods on managed clusters goes into CrashLoopBackOff state and doesn't recover.

Version of all relevant components (if applicable):
OCP: 4.14.0-0.nightly-2023-07-26-132453
ODF: 4.14.0-86
ACM: 2.9.0-62 (quay.io:443/acm-d/acm-custom-registry:2.9.0-DOWNSTREAM-2023-07-26-20-16-55)
Submariner: 0.16.0 (brew.registry.redhat.io/rh-osbs/iib:543072)

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, DRPolicy doesn't reach Validated state

Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
1/1

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deploy 3 OCP 4.14 clusters
2. Install RHACM on hub, import other two clusters and connect using Submariner add-ons with globalnet enabled
3. Deploy ODF 4.14 on both managed clusters
4. Create DRPolicy on hub cluster
5. Observe the pod status


Actual results:
token-exchange-agent pod in CrashLoopBackOff state 

Expected results:
token-exchange-agent pod in Running state 

Additional info:

Pod status:
token-exchange-agent-59d8c546f4-99kcv                             0/1     CrashLoopBackOff   18 (22s ago)   67m    10.135.0.52   compute-0   <none>           <none>

Pod logs:
```
W0727 12:29:39.796683       1 cmd.go:213] Using insecure, self-signed certificates
I0727 12:29:40.021360       1 observer_polling.go:159] Starting file observer
I0727 12:29:40.087867       1 builder.go:262] tokenexchange version -
I0727 12:29:40.261135       1 token_exchanger_agent.go:58] Running "tokenexchange"
W0727 12:29:40.261743       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
W0727 12:29:40.261758       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
I0727 12:29:40.262583       1 common.go:40] Health probes server is running.
I0727 12:29:40.264325       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0727 12:29:40.264342       1 shared_informer.go:273] Waiting for caches to sync for RequestHeaderAuthRequestController
I0727 12:29:40.264371       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0727 12:29:40.264399       1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0727 12:29:40.264397       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0727 12:29:40.264424       1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0727 12:29:40.264706       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-3350406937/tls.crt::/tmp/serving-cert-3350406937/tls.key"
I0727 12:29:40.265177       1 secure_serving.go:210] Serving securely on [::]:8443
I0727 12:29:40.265212       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14bd0d0]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/aggregated_discovery.go:114
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00004b920, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/aggregated_discovery.go:95 +0x6f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0006d8000, 0x15}, {0xc0010ba020, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/aggregated_discovery.go:49 +0x125
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xc000d567b0?)
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:328 +0x3de
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc000d56be0?)
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:203 +0x65
k8s.io/client-go/discovery.ServerGroupsAndResources({0x2f0cae8, 0xc0011a5e30})
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:413 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:376 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc000d56bf8)
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:651 +0x71
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:375 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x2f0cae8?, 0xc0011a5e30?})
	/remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/apiutil/dynamicrestmapper.go:130
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000974b40?, {0x0, 0x0, 0x2a98b4d?})
	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182
sigs.k8s.io/controller-runtime/pkg/client.newClient(0xc000974b40?, {0xc000bbbab0?, {0x0?, 0x0?}, {0x0?, 0x0?}})
	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/client.go:109 +0x1d1
sigs.k8s.io/controller-runtime/pkg/client.New(...)
	/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/client.go:77
github.com/red-hat-storage/odf-multicluster-orchestrator/addons/token-exchange.getClient(0x41416a?)
	/remote-source/app/addons/token-exchange/manager.go:51 +0x29
github.com/red-hat-storage/odf-multicluster-orchestrator/addons/token-exchange.registerHandler({0x7ffc992dab08?, 0xc000a1a4e0?}, 0x1bf08eb000?, 0xc000d571b8?)
	/remote-source/app/addons/token-exchange/secret_exchange_handler_register.go:29 +0x109
github.com/red-hat-storage/odf-multicluster-orchestrator/addons/token-exchange.(*AgentOptions).RunAgent(0xc000cb17a0, {0x2efdc50?, 0xc000cea0a0}, 0xc00102b100)
	/remote-source/app/addons/token-exchange/token_exchanger_agent.go:83 +0x35a
github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerBuilder).Run(0xc000d02120, {0x2efdc50?, 0xc000cea0a0}, 0x0)
	/remote-source/deps/gomod/pkg/mod/github.com/openshift/library-go.0-20230127195720-edf819b079cf/pkg/controller/controllercmd/builder.go:311 +0x15bb
github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).StartController(0xc000bea9c0, {0x2efdc50?, 0xc0009fe000})
	/remote-source/deps/gomod/pkg/mod/github.com/openshift/library-go.0-20230127195720-edf819b079cf/pkg/controller/controllercmd/cmd.go:294 +0x625
github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommandWithContext.func1(0xc000cf6000?, {0x2a79e2f?, 0x4?, 0x4?})
	/remote-source/deps/gomod/pkg/mod/github.com/openshift/library-go.0-20230127195720-edf819b079cf/pkg/controller/controllercmd/cmd.go:137 +0x3eb
github.com/spf13/cobra.(*Command).execute(0xc000cf6000, {0xc000ce89c0, 0x4, 0x4})
	/remote-source/deps/gomod/pkg/mod/github.com/spf13/cobra.1/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x44f3fc0)
	/remote-source/deps/gomod/pkg/mod/github.com/spf13/cobra.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/remote-source/deps/gomod/pkg/mod/github.com/spf13/cobra.1/command.go:968
github.com/red-hat-storage/odf-multicluster-orchestrator/cmd.Execute()
	/remote-source/app/cmd/root.go:22 +0x25
main.main()
	/remote-source/app/main.go:31 +0x17
```
Comment 4 umanga 2023-07-31 11:24:57 UTC 
Looks like the issue is caused by a kubernetes dependency. We are upgrading to a version which has the fix.
Comment 11 errata-xmlrpc 2023-11-08 18:52:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6832