Description of problem: I am trying to deploy the observability components by following the official docs at: https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.1/html/observing_environments/observing-environments Right off the bat, the endpoint-observability-operator pod throws RBAC related errors, see below. Version-Release number of selected component (if applicable): rhacm-2.1 How reproducible: $ oc logs -n open-cluster-management-addon-observability -c endpoint-observability-operator pod/endpoint-observability-operator-85c47d89d6-jrhbs ts=2021-01-06T16:51:41.618753734Z level=info logger=cmd msg="Operator Version: 0.0.1" ts=2021-01-06T16:51:41.618916675Z level=info logger=cmd msg="Go Version: go1.13.15" ts=2021-01-06T16:51:41.618934635Z level=info logger=cmd msg="Go OS/Arch: linux/amd64" ts=2021-01-06T16:51:41.618948155Z level=info logger=cmd msg="Version of operator-sdk: v0.17.0" ts=2021-01-06T16:51:41.619768048Z level=info logger=leader msg="Trying to become the leader." ts=2021-01-06T16:51:41.619819329Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability ts=2021-01-06T16:51:44.474638784Z level=debug logger=k8sutil msg="Found podname" Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs ts=2021-01-06T16:51:44.482126005Z level=debug logger=k8sutil msg="Found Pod" Pod.Namespace=open-cluster-management-addon-observability Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs ts=2021-01-06T16:51:44.486986474Z level=info logger=leader msg="No pre-existing lock was found." ts=2021-01-06T16:51:44.491381953Z level=info logger=leader msg="Became the leader." ts=2021-01-06T16:51:45.15507388Z level=info logger=controller-runtime.metrics msg="metrics server is starting to listen" addr=0.0.0.0:8383 ts=2021-01-06T16:51:45.155380821Z level=info logger=cmd msg="Registering Components." W0106 16:51:45.155504 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. W0106 16:51:45.156648 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. ts=2021-01-06T16:51:45.156962398Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability ts=2021-01-06T16:51:45.157035208Z level=debug logger=kubemetrics msg="Starting collecting operator types" ts=2021-01-06T16:51:45.157043998Z level=debug logger=kubemetrics msg="Generating metric families" apiVersion=observability.open-cluster-management.io/v1beta1 kind=MultiClusterObservability ts=2021-01-06T16:51:48.009957137Z level=debug logger=kubemetrics msg="Generating metric families" apiVersion=observability.open-cluster-management.io/v1beta1 kind=ObservabilityAddon ts=2021-01-06T16:51:50.863544966Z level=debug logger=kubemetrics msg="Starting serving custom resource metrics" ts=2021-01-06T16:51:53.716314723Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability ts=2021-01-06T16:51:53.716432843Z level=debug logger=k8sutil msg="Found podname" Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs ts=2021-01-06T16:51:53.720465009Z level=debug logger=k8sutil msg="Found Pod" Pod.Namespace=open-cluster-management-addon-observability Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs ts=2021-01-06T16:51:53.725064168Z level=info logger=cmd msg="Could not create metrics Service" error="failed to initialize service object for metrics: appliedmanifestworks.work.open-cluster-management.io \"d74ef2c1b26c2ab1be897c5489d25c9f6ac4f9b01cbfefc8a4ece6aac56be584-endpoint-observability-work\" is forbidden: User \"system:serviceaccount:open-cluster-management-addon-observability:endpoint-observability-operator-sa\" cannot get resource \"appliedmanifestworks\" in API group \"work.open-cluster-management.io\" at the cluster scope" Additional info: I "fixed" it by providing the operator with the full permissions to Kubernetes APIs by creating these resources: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: endpoint-observability-operator-rb-fix roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: endpoint-observability-operator-fix subjects: - kind: ServiceAccount name: endpoint-observability-operator-sa namespace: open-cluster-management-addon-observability apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: endpoint-observability-operator-fix rules: - apiGroups: - '*' resources: - '*' verbs: - '*'
This has been verified in ACM 2.2.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Advanced Cluster Management for Kubernetes version 2.2 images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:0729