1894778 – Managed Cluster in RHACM stays in Pending Import state

Bug 1894778 - Managed Cluster in RHACM stays in Pending Import state

Summary: Managed Cluster in RHACM stays in Pending Import state

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Advanced Cluster Management for Kubernetes
Classification:	Red Hat
Component:	Cluster Lifecycle
Sub Component:
Version:	rhacm-2.1
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	rhacm-2.1.3
Assignee:	Hao Liu
QA Contact:	juhsu
Docs Contact:	Christopher Dawson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-11-05 02:43 UTC by Fran Kemp
Modified:	2023-09-15 00:50 UTC (History)
CC List:	4 users (show)
Fixed In Version:	rhacm-2.1.3
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-02-17 18:19:07 UTC
Target Upstream Version:
Embargoed:
Flags:	gghezzo: rhacm-2.1.z+ gghezzo: rhacm-2.2+

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	open-cluster-management backlog issues 6856	0	None	None	None	2021-02-22 14:18:42 UTC
Red Hat Product Errata	RHSA-2021:0607	0	None	None	None	2021-02-17 18:19:24 UTC

Description Fran Kemp 2020-11-05 02:43:11 UTC

Description of problem:

Installed RHACM V2.0.4 on an OpenShift V4.5 cluster on the IBM Cloud.

Attempting to import 2 Kubernetes clusters - both on the IBM Cloud.  Both clusters are getting the same error - they stay permanently in the Pending Import state.

There are 2 errors in the klusterlet-registration-agent pod:
E1104 22:07:30.421104       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:07:30.426849       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority

The log file errors are indicating certificate signer issues - I'm not sure where to put the signing certs to get this to work.  I though it might go in the customCAConfigmap field when the MultiClusterHub is created, but couldn't find any references to this field in the documentation.  I suspect it might go in the YAML file when the cluster is imported but I'm not sure how to do that.

This error is similar to Bug 1860233 - except there is a different error in the pod log.



Version-Release number of selected component (if applicable):
V2.0.4

How reproducible:
Import Existing Cluster 

Steps to Reproduce:
1.Install MutlClusterHub
2.Import an Existing K8S cluster
3.

Actual results:
Imported Cluster stays in Pening state

Expected results:
Imported Cluster should be managed

Additional info:

Here is the full pod log for the klusterlet-registration-agent pod:

W1104 22:07:28.852209       1 cmd.go:204] Using insecure, self-signed certificates
I1104 22:07:29.533640       1 observer_polling.go:159] Starting file observer
W1104 22:07:29.556258       1 builder.go:206] unable to get owner reference (falling back to namespace): pods is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-registration-sa" cannot list resource "pods" in API group "" in the namespace "open-cluster-management-agent"
I1104 22:07:29.556524       1 builder.go:233] registration-agent version 1b8d096-1b8d0968819d62ebab2d32d2bf02e237ad7e9052
I1104 22:07:30.239529       1 leaderelection.go:242] attempting to acquire leader lease  open-cluster-management-agent/registration-agent-lock...
I1104 22:07:30.239819       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1104 22:07:30.239848       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1104 22:07:30.239828       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1104 22:07:30.239901       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1104 22:07:30.240371       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/serving-cert-621297144/tls.crt::/tmp/serving-cert-621297144/tls.key
I1104 22:07:30.240862       1 secure_serving.go:178] Serving securely on [::]:8443
I1104 22:07:30.240905       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1104 22:07:30.276738       1 leaderelection.go:252] successfully acquired lease open-cluster-management-agent/registration-agent-lock
I1104 22:07:30.277210       1 spokeagent.go:93] Cluster name is "ic-dal10-k8s" and agent name is "rxk49"
I1104 22:07:30.277288       1 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"open-cluster-management-agent", Name:"registration-agent-lock", UID:"6a87a317-00d9-4896-b56d-5a0b85f36e47", APIVersion:"v1", ResourceVersion:"27271733", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' afddeee2-188a-492b-a43d-620b08e3d8ee became leader
I1104 22:07:30.289443       1 shared_informer.go:223] Waiting for caches to sync for ManagedClusterCreatingController
I1104 22:07:30.289473       1 shared_informer.go:230] Caches are synced for ManagedClusterCreatingController
I1104 22:07:30.289485       1 base_controller.go:79] Starting #1 worker of ManagedClusterCreatingController controller ...
I1104 22:07:30.289972       1 spokeagent.go:165] Waiting for hub client config and managed cluster to be ready
I1104 22:07:30.290314       1 shared_informer.go:223] Waiting for caches to sync for BootstrapClientCertForHubController
I1104 22:07:30.340054       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1104 22:07:30.340169       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E1104 22:07:30.421104       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:07:30.426849       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:30.521764       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:30.623070       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:30.723967       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:30.850700       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:31.006410       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:31.368177       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:31.792506       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:31.935315       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:07:32.504949       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:33.890699       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:35.199767       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:07:36.537363       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:41.470530       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:07:41.736593       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:07:48.661085       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:07:52.053623       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:08:07.862805       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:08:12.634867       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:08:53.689770       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:08:55.983717       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:09:50.002701       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:10:15.701333       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:10:35.324065       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:11:15.109666       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:12:12.880478       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:12:59.628304       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:13:03.177137       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:13:47.866362       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:14:19.139372       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:15:05.218822       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:15:48.958111       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:16:26.591603       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:17:22.663174       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:18:16.883010       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:18:27.413124       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:19:13.974085       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:19:48.869594       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:20:45.275723       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:21:25.569283       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:22:05.577900       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:23:02.395117       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:23:56.028572       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:24:31.839505       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:25:20.148604       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:26:03.970698       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:26:42.291618       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:27:17.235576       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:28:05.419091       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:28:54.712352       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:29:22.885204       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:29:37.040123       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:30:34.313031       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:31:32.525031       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:32:04.554456       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:32:41.926737       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:33:25.394805       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:34:15.643179       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:35:02.318387       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:35:42.714576       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:36:32.081603       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:37:32.091660       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:38:23.634329       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:39:08.171897       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:39:47.049835       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:40:20.438343       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:41:16.841113       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:42:10.257942       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:42:47.491233       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:43:20.003790       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:43:51.531724       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:44:46.358404       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:45:37.173349       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:46:02.970253       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 22:46:31.845949       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:47:23.574127       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:48:00.852922       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:48:51.028040       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:49:45.599926       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:50:32.528536       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:51:19.930355       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:52:15.230392       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:53:08.834524       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:53:53.639715       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:54:31.424582       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:55:17.774540       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:55:57.625421       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:56:41.209252       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:57:24.324289       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:58:00.436722       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:58:49.295498       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 22:59:38.951313       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:00:29.651793       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:01:03.201892       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:01:41.775125       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:02:25.769825       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:02:43.056012       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 23:03:14.941421       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:04:03.413943       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:04:37.704491       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:05:09.927608       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:06:02.221602       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:06:44.115928       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:07:30.371958       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 23:07:40.666546       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:08:16.164282       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:09:05.712024       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:09:48.598769       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:10:40.816330       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:11:18.858403       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:11:54.142698       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:12:49.270706       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:13:41.214469       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:14:26.056659       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:15:11.094863       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:16:04.857881       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:16:59.124925       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:17:57.996860       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:18:36.424139       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:19:11.835741       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:19:23.147248       1 base_controller.go:219] "ManagedClusterCreatingController" controller failed to sync "key", err: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/cluster.open-cluster-management.io/v1/managedclusters/ic-dal10-k8s: x509: certificate signed by unknown authority
E1104 23:20:10.122989       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority
E1104 23:21:05.387552       1 reflector.go:178] k8s.io/client-go.3/tools/cache/reflector.go:125: Failed to list *v1beta1.CertificateSigningRequest: Get https://c100-e.us-east.containers.cloud.ibm.com:32310/apis/certificates.k8s.io/v1beta1/certificatesigningrequests?limit=500&resourceVersion=0: x509: certificate signed by unknown authority

Comment 1 Fran Kemp 2020-11-07 02:48:09 UTC

Uninstalled RHACM V2.0.4 and upgraded to V2.1

Got the same error when trying to import the K8s clusters.

Forgot to include this initially, but the K8s clusters I'm trying to import are V1.18.10 and V1.19.3.
Same error on each.

Comment 2 Mike Ng 2020-11-10 20:46:06 UTC

G2Bsync 724943764 comment 
 TheRealHaoLiu Tue, 10 Nov 2020 20:19:50 UTC 
 G2Bsync
the root cause of this problem is that the managed cluster import controller is not able to determine the right CA certificate that the agent should use to communicate back to the hub

in an regular openshift cluster there are two scenarios for configuring the kube API server certificate
1. custom CA cert configured in the `kubeapiserver` resource
2. self signed CA cert generated by openshift if no custom CA cert is configured

the managed cluster import controller uses `kubeapiserver` resource to determine which cert to use for taking to the hub 

IBM ROKS cluster uses a signed custom CA certificate however due to the unique nature of the master nodes (that it is running as pod in another cluster) the kubeapiserver controller is disabled and the resource was not configured to contain the CA cert information 

from managed cluster import controller's perspective it seems as if the hub does not have an custom CA cert (which is incorrect in this case but given then information that it have access to it have no way of knowing that) and uses the self sign CA for the agent to communicate with the hub

the current proposed solution to IBM is that ROKS should configure `kubeapiserver` resource with the proper information (even tho the resource itself is not being acted upon by the kube apiserver controller) and thus give managed cluster import controller a way to determine the correct CA

Comment 3 Fran Kemp 2020-11-11 04:27:55 UTC

Thanks for the explanation.  Can the kubeapiserver resource be added manually?

Comment 4 gajan@jp.ibm.com 2020-11-11 14:14:32 UTC

Hi,

I tried installing RHACM2.1 on OpenShift on AWS.

Server Version: 4.5.11
Kubernetes Version: v1.18.3+b0068a8

I see above issue both in local-cluster on hub, as well as on imported cluster (another Openshift cluster on AWS)

Do you have any suggestion for resolving it on Openshift cluster on AWS ?

Thank you.
...................................................................................................................................................
$oc logs cluster-manager-registration-controller-68cb65bbf8-2pjnz -n open-cluster-management-hub
I1111 13:56:57.607090   25114 request.go:621] Throttling request took 1.177157826s, request: GET:https://api.ma4kdev4.openshiftv4test.com:6443/apis/kibana.k8s.elastic.co/v1?timeout=32s
W1111 10:12:49.644615       1 cmd.go:204] Using insecure, self-signed certificates
I1111 10:12:49.943925       1 observer_polling.go:159] Starting file observer
W1111 10:12:49.956066       1 builder.go:207] unable to get owner reference (falling back to namespace): pods is forbidden: User "system:serviceaccount:open-cluster-management-hub:cluster-manager-registration-controller-sa" cannot list resource "pods" in API group "" in the namespace "open-cluster-management-hub"
I1111 10:12:49.956620       1 builder.go:238] registration-controller version v0.0.0-unknown-
W1111 10:12:50.488461       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
W1111 10:12:50.488809       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
I1111 10:12:50.489536       1 leaderelection.go:243] attempting to acquire leader lease  open-cluster-management-hub/registration-controller-lock...
I1111 10:12:50.492694       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1111 10:12:50.492720       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1111 10:12:50.492742       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1111 10:12:50.492753       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1111 10:12:50.492744       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1111 10:12:50.492803       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1111 10:12:50.493039       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/serving-cert-831792488/tls.crt::/tmp/serving-cert-831792488/tls.key
I1111 10:12:50.493239       1 secure_serving.go:197] Serving securely on [::]:8443

...................................................................................................................................................
$oc logs klusterlet-6dcd6d6bd9-hwflx -n open-cluster-management-agent

W1111 10:15:12.949920       1 cmd.go:204] Using insecure, self-signed certificates
I1111 10:15:13.093515       1 observer_polling.go:159] Starting file observer
W1111 10:15:13.102888       1 builder.go:207] unable to get owner reference (falling back to namespace): pods is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet" cannot list resource "pods" in API group "" in the namespace "open-cluster-management-agent"
I1111 10:15:13.102996       1 builder.go:238] klusterlet version v0.0.0-unknown-
W1111 10:15:13.335509       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
W1111 10:15:13.335523       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
I1111 10:15:13.336233       1 leaderelection.go:243] attempting to acquire leader lease  open-cluster-management-agent/klusterlet-lock...
I1111 10:15:13.339570       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1111 10:15:13.339575       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1111 10:15:13.339593       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1111 10:15:13.339600       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1111 10:15:13.339637       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1111 10:15:13.339597       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1111 10:15:13.339830       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/serving-cert-630372349/tls.crt::/tmp/serving-cert-630372349/tls.key
I1111 10:15:13.340150       1 secure_serving.go:197] Serving securely on [::]:8443

...................................................................................................................................................

Comment 5 Fran Kemp 2020-11-11 18:03:20 UTC

Hi Mike,

Is there any solution for this issue?

According to https://access.redhat.com/articles/5248271, the version of our Hub Cluster (4.5) and the versions of our IBM Cloud Kubernetes Service clusters (1.18 and 1.19) are both officially supported for management.

Fran

Comment 6 Mike Ng 2020-11-18 14:37:06 UTC

G2Bsync 729717693 comment 
 juliana-hsu Wed, 18 Nov 2020 14:32:15 UTC 
 G2Bsync   Hao Liu:  ROKS hub is not supported.  ROKS spoke is supported

Comment 7 Fran Kemp 2020-11-20 15:59:27 UTC

Thanks - missed that distinction earlier.  

Any plans to support the ROKS hub?

Comment 14 errata-xmlrpc 2021-02-17 18:19:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.1.3 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0607

Comment 15 Red Hat Bugzilla 2023-09-15 00:50:43 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days

Note You need to log in before you can comment on or make changes to this bug.