Thanks for report this issue. That is an known issue of ACM 2.1. Could you follow this step to resolve your issue - https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.1/html/troubleshooting/troubleshooting#symptom-observability-invalid-certification
The pod multicluster-observability-operator-<pod_name> does not exist for the customer
multicluster-observability-operator-<pod_name> is in ACM namespace. for example: open-cluster-management if no changes made.
@tuado How can I access the must-gather information? I did not see a link in the case. Thanks.
@rspagnol I can access the logs and found that `ts=2021-02-27T16:32:46` said that error loading config file \"/spoke/hub-kubeconfig/kubeconfig `ts=2021-02-24T20:53:57` said that certificate signed by unknown authority from the description, I can see the certificate signed by unknown authority error at 2021-03-01 the customer environment has. It cannot connect with API server. I remembered we have the similar issue logged in server foundation. @jqiu do we have document or fix for this issue? Thanks.
@jayoung Can we setup a meeting or slack channel to discuss? We cannot reproduce it in local environment. Thanks.
In Mar 26th, we have an online meeting for debug. Finally we made this flow work with custom ca cert. In fact we are following steps here: https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.1/html/troubleshooting/troubleshooting#symptom-observability-invalid-certification But there are two things required attentions: 1. In MCO CR, if we set annotation mco-pause as true, observability will not be enabled in the newly-imported cluster. If we delete this annotation, the changes we made in the workaround will be reverted. So each time when users remove the mco-pause annotation and add it back, need to manually apply the steps in the workaround again. 2. In the kubeconfig secret which needs to be patched in manifestwork, it contains ca cert and token. For every imported clusters, the ca cert is the same one, but the token is different. So when applying the fix, the same kubeconfig cannot be applied to every imported clusters, each cluster should patched its' own version.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Advanced Cluster Management 2.1.8 bug fix and container updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2540