Bug 1894778
| Summary: | Managed Cluster in RHACM stays in Pending Import state | ||
|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | Fran Kemp <francis.kemp> |
| Component: | Cluster Lifecycle | Assignee: | Hao Liu <haoli> |
| Status: | CLOSED ERRATA | QA Contact: | juhsu |
| Severity: | high | Docs Contact: | Christopher Dawson <cdawson> |
| Priority: | unspecified | ||
| Version: | rhacm-2.1 | CC: | danclark, gajan, gghezzo, haoli |
| Target Milestone: | --- | Flags: | gghezzo:
rhacm-2.1.z+
gghezzo: rhacm-2.2+ |
| Target Release: | rhacm-2.1.3 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | rhacm-2.1.3 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-17 18:19:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Fran Kemp
2020-11-05 02:43:11 UTC
Uninstalled RHACM V2.0.4 and upgraded to V2.1 Got the same error when trying to import the K8s clusters. Forgot to include this initially, but the K8s clusters I'm trying to import are V1.18.10 and V1.19.3. Same error on each. G2Bsync 724943764 comment TheRealHaoLiu Tue, 10 Nov 2020 20:19:50 UTC G2Bsync the root cause of this problem is that the managed cluster import controller is not able to determine the right CA certificate that the agent should use to communicate back to the hub in an regular openshift cluster there are two scenarios for configuring the kube API server certificate 1. custom CA cert configured in the `kubeapiserver` resource 2. self signed CA cert generated by openshift if no custom CA cert is configured the managed cluster import controller uses `kubeapiserver` resource to determine which cert to use for taking to the hub IBM ROKS cluster uses a signed custom CA certificate however due to the unique nature of the master nodes (that it is running as pod in another cluster) the kubeapiserver controller is disabled and the resource was not configured to contain the CA cert information from managed cluster import controller's perspective it seems as if the hub does not have an custom CA cert (which is incorrect in this case but given then information that it have access to it have no way of knowing that) and uses the self sign CA for the agent to communicate with the hub the current proposed solution to IBM is that ROKS should configure `kubeapiserver` resource with the proper information (even tho the resource itself is not being acted upon by the kube apiserver controller) and thus give managed cluster import controller a way to determine the correct CA Thanks for the explanation. Can the kubeapiserver resource be added manually? Hi, I tried installing RHACM2.1 on OpenShift on AWS. Server Version: 4.5.11 Kubernetes Version: v1.18.3+b0068a8 I see above issue both in local-cluster on hub, as well as on imported cluster (another Openshift cluster on AWS) Do you have any suggestion for resolving it on Openshift cluster on AWS ? Thank you. ................................................................................................................................................... $oc logs cluster-manager-registration-controller-68cb65bbf8-2pjnz -n open-cluster-management-hub I1111 13:56:57.607090 25114 request.go:621] Throttling request took 1.177157826s, request: GET:https://api.ma4kdev4.openshiftv4test.com:6443/apis/kibana.k8s.elastic.co/v1?timeout=32s W1111 10:12:49.644615 1 cmd.go:204] Using insecure, self-signed certificates I1111 10:12:49.943925 1 observer_polling.go:159] Starting file observer W1111 10:12:49.956066 1 builder.go:207] unable to get owner reference (falling back to namespace): pods is forbidden: User "system:serviceaccount:open-cluster-management-hub:cluster-manager-registration-controller-sa" cannot list resource "pods" in API group "" in the namespace "open-cluster-management-hub" I1111 10:12:49.956620 1 builder.go:238] registration-controller version v0.0.0-unknown- W1111 10:12:50.488461 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W1111 10:12:50.488809 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I1111 10:12:50.489536 1 leaderelection.go:243] attempting to acquire leader lease open-cluster-management-hub/registration-controller-lock... I1111 10:12:50.492694 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I1111 10:12:50.492720 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController I1111 10:12:50.492742 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file I1111 10:12:50.492753 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I1111 10:12:50.492744 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I1111 10:12:50.492803 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I1111 10:12:50.493039 1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/serving-cert-831792488/tls.crt::/tmp/serving-cert-831792488/tls.key I1111 10:12:50.493239 1 secure_serving.go:197] Serving securely on [::]:8443 ................................................................................................................................................... $oc logs klusterlet-6dcd6d6bd9-hwflx -n open-cluster-management-agent W1111 10:15:12.949920 1 cmd.go:204] Using insecure, self-signed certificates I1111 10:15:13.093515 1 observer_polling.go:159] Starting file observer W1111 10:15:13.102888 1 builder.go:207] unable to get owner reference (falling back to namespace): pods is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet" cannot list resource "pods" in API group "" in the namespace "open-cluster-management-agent" I1111 10:15:13.102996 1 builder.go:238] klusterlet version v0.0.0-unknown- W1111 10:15:13.335509 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W1111 10:15:13.335523 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I1111 10:15:13.336233 1 leaderelection.go:243] attempting to acquire leader lease open-cluster-management-agent/klusterlet-lock... I1111 10:15:13.339570 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file I1111 10:15:13.339575 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I1111 10:15:13.339593 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I1111 10:15:13.339600 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I1111 10:15:13.339637 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController I1111 10:15:13.339597 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I1111 10:15:13.339830 1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/serving-cert-630372349/tls.crt::/tmp/serving-cert-630372349/tls.key I1111 10:15:13.340150 1 secure_serving.go:197] Serving securely on [::]:8443 ................................................................................................................................................... Hi Mike, Is there any solution for this issue? According to https://access.redhat.com/articles/5248271, the version of our Hub Cluster (4.5) and the versions of our IBM Cloud Kubernetes Service clusters (1.18 and 1.19) are both officially supported for management. Fran G2Bsync 729717693 comment juliana-hsu Wed, 18 Nov 2020 14:32:15 UTC G2Bsync Hao Liu: ROKS hub is not supported. ROKS spoke is supported Thanks - missed that distinction earlier. Any plans to support the ROKS hub? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.1.3 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0607 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |