Description of problem: Customer has installed a RHACM 2.1 hub cluster with a custom CA/Cert for the API. RHACM component addons work fine however when the observability component is added the following error is generate: oc logs endpoint-observability-operator-6fdc995b8f-8t8rb -c endpoint-observability-operator ts=2020-12-10T14:38:09.90476176Z level=info logger=cmd msg="Operator Version: 0.0.1" ts=2020-12-10T14:38:09.904788522Z level=info logger=cmd msg="Go Version: go1.13.15" ts=2020-12-10T14:38:09.904793685Z level=info logger=cmd msg="Go OS/Arch: linux/amd64" ts=2020-12-10T14:38:09.904797148Z level=info logger=cmd msg="Version of operator-sdk: v0.17.0" ts=2020-12-10T14:38:09.90523669Z level=info logger=leader msg="Trying to become the leader." ts=2020-12-10T14:38:09.905266302Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability ts=2020-12-10T14:38:13.714129298Z level=debug logger=k8sutil msg="Found podname" Pod.Name=endpoint-observability-operator-6fdc995b8f-8t8rb ts=2020-12-10T14:38:13.727655349Z level=debug logger=k8sutil msg="Found Pod" Pod.Namespace=open-cluster-management-addon-observability Pod.Name=endpoint-observability-operator-6fdc995b8f-8t8rb ts=2020-12-10T14:38:13.73263813Z level=info logger=leader msg="Found existing lock with my name. I was likely restarted." ts=2020-12-10T14:38:13.732664067Z level=info logger=leader msg="Continuing as the leader." ts=2020-12-10T14:38:13.763256787Z level=error logger=controller-runtime.manager msg="Failed to get API Group-Resources" error="Get https://api.vcptoisp001t.mycustomer.xxx:6443/api?timeout=32s: x509: certificate signed by unknown authority" stacktrace="github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/app/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/manager/manager.go:241\nmain.main\n\t/remote-source/app/cmd/manager/main.go:126\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:203" ts=2020-12-10T14:38:13.763345263Z level=error logger=cmd msg= error="Get https://api.vcptoisp001t.mycustomer.xxx:6443/api?timeout=32s: x509: certificate signed by unknown authority" stacktrace="github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/app/vendor/github.com/go-logr/zapr/zapr.go:128\nmain.main\n\t/remote-source/app/cmd/manager/main.go:128\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:203" Version-Release number of selected component (if applicable): 2.1 How reproducible: 100% Steps to Reproduce: 1. Install OCP Cluster with custom CA/Cert 2. Install RHACM on OCP cluster 3. Install Observability component on hub cluster Actual results: x509: certificate signed by unknown authority Expected results: API should be accessible Additional info:
Thanks @bschmaus for trying the observability feature in ACM 2.1. We are working on fixing this issue ASAP. Here are steps to workaround this issue: 1. Pause the multiclusterobservability operator ``` oc edit mco observability ``` to add `mco-pause: "true"`. it looks like: ``` apiVersion: observability.open-cluster-management.io/v1beta1 kind: MultiClusterObservability metadata: annotations: mco-pause: "true" ``` 2. Modify manifestwork `endpoint-observability-work` in managed cluster namespace. for example: local-cluster. a. save `kubeconfig` to modify `oc get manifestwork -n local-cluster -oyaml endpoint-observability-work | grep kubeconfig: | awk '{print $2}' | base64 -d > kubeconfig` b. update `certificate-authority-data` with the customized one. the value is `cat ca-bundle.crt | base64 -w0 ; echo` as mentioned in https://two-oes.medium.com/openshift-4-custom-certificate-for-the-cluster-api-bf2565f60bb9 c. replace the value of `kubeconfig:` in `endpoint-observability-work` with `cat kubeconfig | base64 -w0; echo` 3. `oc delete secret hub-kube-config -n open-cluster-management-addon-observability` and wait for new hub-kube-config secret created. Then restart `endpoint-observability-operator`. Let me know if it works in your environment or not. Thank you !
We have applied the workaround ad is working with multiple managed clusters. Take into consideration that is easier to start with the local-cluster only( no managed clusters already add ) and repeat all the steps for every managed cluster add from the hub, we suggest in our experience to add and patch one-by-one. regards
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.1.3 security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:0607