Description of problem (please be detailed as possible and provide log snippests): using OCP 4.14, while configuring RDR setup, after creating DRPolicy on hub cluster, observed that token-exchange-agent pods on managed clusters goes into CrashLoopBackOff state and doesn't recover. Version of all relevant components (if applicable): OCP: 4.14.0-0.nightly-2023-07-26-132453 ODF: 4.14.0-86 ACM: 2.9.0-62 (quay.io:443/acm-d/acm-custom-registry:2.9.0-DOWNSTREAM-2023-07-26-20-16-55) Submariner: 0.16.0 (brew.registry.redhat.io/rh-osbs/iib:543072) Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes, DRPolicy doesn't reach Validated state Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? 1/1 Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Deploy 3 OCP 4.14 clusters 2. Install RHACM on hub, import other two clusters and connect using Submariner add-ons with globalnet enabled 3. Deploy ODF 4.14 on both managed clusters 4. Create DRPolicy on hub cluster 5. Observe the pod status Actual results: token-exchange-agent pod in CrashLoopBackOff state Expected results: token-exchange-agent pod in Running state Additional info: Pod status: token-exchange-agent-59d8c546f4-99kcv 0/1 CrashLoopBackOff 18 (22s ago) 67m 10.135.0.52 compute-0 <none> <none> Pod logs: ``` W0727 12:29:39.796683 1 cmd.go:213] Using insecure, self-signed certificates I0727 12:29:40.021360 1 observer_polling.go:159] Starting file observer I0727 12:29:40.087867 1 builder.go:262] tokenexchange version - I0727 12:29:40.261135 1 token_exchanger_agent.go:58] Running "tokenexchange" W0727 12:29:40.261743 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0727 12:29:40.261758 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0727 12:29:40.262583 1 common.go:40] Health probes server is running. I0727 12:29:40.264325 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I0727 12:29:40.264342 1 shared_informer.go:273] Waiting for caches to sync for RequestHeaderAuthRequestController I0727 12:29:40.264371 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0727 12:29:40.264399 1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0727 12:29:40.264397 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0727 12:29:40.264424 1 shared_informer.go:273] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0727 12:29:40.264706 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-3350406937/tls.crt::/tmp/serving-cert-3350406937/tls.key" I0727 12:29:40.265177 1 secure_serving.go:210] Serving securely on [::]:8443 I0727 12:29:40.265212 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x14bd0d0] goroutine 1 [running]: k8s.io/client-go/discovery.convertAPIResource(...) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/aggregated_discovery.go:114 k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00004b920, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/aggregated_discovery.go:95 +0x6f0 k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0006d8000, 0x15}, {0xc0010ba020, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/aggregated_discovery.go:49 +0x125 k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xc000d567b0?) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:328 +0x3de k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc000d56be0?) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:203 +0x65 k8s.io/client-go/discovery.ServerGroupsAndResources({0x2f0cae8, 0xc0011a5e30}) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:413 +0x59 k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1() /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:376 +0x25 k8s.io/client-go/discovery.withRetries(0x2, 0xc000d56bf8) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:651 +0x71 k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/discovery/discovery_client.go:375 +0x3a k8s.io/client-go/restmapper.GetAPIGroupResources({0x2f0cae8?, 0xc0011a5e30?}) /remote-source/deps/gomod/pkg/mod/k8s.io/client-go.3/restmapper/discovery.go:148 +0x42 sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1() /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25 sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/apiutil/dynamicrestmapper.go:130 sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000974b40?, {0x0, 0x0, 0x2a98b4d?}) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182 sigs.k8s.io/controller-runtime/pkg/client.newClient(0xc000974b40?, {0xc000bbbab0?, {0x0?, 0x0?}, {0x0?, 0x0?}}) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/client.go:109 +0x1d1 sigs.k8s.io/controller-runtime/pkg/client.New(...) /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/client/client.go:77 github.com/red-hat-storage/odf-multicluster-orchestrator/addons/token-exchange.getClient(0x41416a?) /remote-source/app/addons/token-exchange/manager.go:51 +0x29 github.com/red-hat-storage/odf-multicluster-orchestrator/addons/token-exchange.registerHandler({0x7ffc992dab08?, 0xc000a1a4e0?}, 0x1bf08eb000?, 0xc000d571b8?) /remote-source/app/addons/token-exchange/secret_exchange_handler_register.go:29 +0x109 github.com/red-hat-storage/odf-multicluster-orchestrator/addons/token-exchange.(*AgentOptions).RunAgent(0xc000cb17a0, {0x2efdc50?, 0xc000cea0a0}, 0xc00102b100) /remote-source/app/addons/token-exchange/token_exchanger_agent.go:83 +0x35a github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerBuilder).Run(0xc000d02120, {0x2efdc50?, 0xc000cea0a0}, 0x0) /remote-source/deps/gomod/pkg/mod/github.com/openshift/library-go.0-20230127195720-edf819b079cf/pkg/controller/controllercmd/builder.go:311 +0x15bb github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).StartController(0xc000bea9c0, {0x2efdc50?, 0xc0009fe000}) /remote-source/deps/gomod/pkg/mod/github.com/openshift/library-go.0-20230127195720-edf819b079cf/pkg/controller/controllercmd/cmd.go:294 +0x625 github.com/openshift/library-go/pkg/controller/controllercmd.(*ControllerCommandConfig).NewCommandWithContext.func1(0xc000cf6000?, {0x2a79e2f?, 0x4?, 0x4?}) /remote-source/deps/gomod/pkg/mod/github.com/openshift/library-go.0-20230127195720-edf819b079cf/pkg/controller/controllercmd/cmd.go:137 +0x3eb github.com/spf13/cobra.(*Command).execute(0xc000cf6000, {0xc000ce89c0, 0x4, 0x4}) /remote-source/deps/gomod/pkg/mod/github.com/spf13/cobra.1/command.go:920 +0x847 github.com/spf13/cobra.(*Command).ExecuteC(0x44f3fc0) /remote-source/deps/gomod/pkg/mod/github.com/spf13/cobra.1/command.go:1044 +0x3bd github.com/spf13/cobra.(*Command).Execute(...) /remote-source/deps/gomod/pkg/mod/github.com/spf13/cobra.1/command.go:968 github.com/red-hat-storage/odf-multicluster-orchestrator/cmd.Execute() /remote-source/app/cmd/root.go:22 +0x25 main.main() /remote-source/app/main.go:31 +0x17 ```
Looks like the issue is caused by a kubernetes dependency. We are upgrading to a version which has the fix.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6832