Bug 1913447

Description of problem:

I am trying to deploy the observability components by following the official docs at:
https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.1/html/observing_environments/observing-environments

Right off the bat, the endpoint-observability-operator pod throws RBAC related errors, see below.

Version-Release number of selected component (if applicable):

rhacm-2.1

How reproducible:

$ oc logs -n open-cluster-management-addon-observability -c endpoint-observability-operator pod/endpoint-observability-operator-85c47d89d6-jrhbs
ts=2021-01-06T16:51:41.618753734Z level=info logger=cmd msg="Operator Version: 0.0.1"
ts=2021-01-06T16:51:41.618916675Z level=info logger=cmd msg="Go Version: go1.13.15"
ts=2021-01-06T16:51:41.618934635Z level=info logger=cmd msg="Go OS/Arch: linux/amd64"
ts=2021-01-06T16:51:41.618948155Z level=info logger=cmd msg="Version of operator-sdk: v0.17.0"
ts=2021-01-06T16:51:41.619768048Z level=info logger=leader msg="Trying to become the leader."
ts=2021-01-06T16:51:41.619819329Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability
ts=2021-01-06T16:51:44.474638784Z level=debug logger=k8sutil msg="Found podname" Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs
ts=2021-01-06T16:51:44.482126005Z level=debug logger=k8sutil msg="Found Pod" Pod.Namespace=open-cluster-management-addon-observability Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs
ts=2021-01-06T16:51:44.486986474Z level=info logger=leader msg="No pre-existing lock was found."
ts=2021-01-06T16:51:44.491381953Z level=info logger=leader msg="Became the leader."
ts=2021-01-06T16:51:45.15507388Z level=info logger=controller-runtime.metrics msg="metrics server is starting to listen" addr=0.0.0.0:8383
ts=2021-01-06T16:51:45.155380821Z level=info logger=cmd msg="Registering Components."
W0106 16:51:45.155504       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0106 16:51:45.156648       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ts=2021-01-06T16:51:45.156962398Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability
ts=2021-01-06T16:51:45.157035208Z level=debug logger=kubemetrics msg="Starting collecting operator types"
ts=2021-01-06T16:51:45.157043998Z level=debug logger=kubemetrics msg="Generating metric families" apiVersion=observability.open-cluster-management.io/v1beta1 kind=MultiClusterObservability
ts=2021-01-06T16:51:48.009957137Z level=debug logger=kubemetrics msg="Generating metric families" apiVersion=observability.open-cluster-management.io/v1beta1 kind=ObservabilityAddon
ts=2021-01-06T16:51:50.863544966Z level=debug logger=kubemetrics msg="Starting serving custom resource metrics"
ts=2021-01-06T16:51:53.716314723Z level=debug logger=k8sutil msg="Found namespace" Namespace=open-cluster-management-addon-observability
ts=2021-01-06T16:51:53.716432843Z level=debug logger=k8sutil msg="Found podname" Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs
ts=2021-01-06T16:51:53.720465009Z level=debug logger=k8sutil msg="Found Pod" Pod.Namespace=open-cluster-management-addon-observability Pod.Name=endpoint-observability-operator-85c47d89d6-jrhbs
ts=2021-01-06T16:51:53.725064168Z level=info logger=cmd msg="Could not create metrics Service" error="failed to initialize service object for metrics: appliedmanifestworks.work.open-cluster-management.io \"d74ef2c1b26c2ab1be897c5489d25c9f6ac4f9b01cbfefc8a4ece6aac56be584-endpoint-observability-work\" is forbidden: User \"system:serviceaccount:open-cluster-management-addon-observability:endpoint-observability-operator-sa\" cannot get resource \"appliedmanifestworks\" in API group \"work.open-cluster-management.io\" at the cluster scope"

Additional info:

I "fixed" it by providing the operator with the full permissions to Kubernetes APIs by creating these resources:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: endpoint-observability-operator-rb-fix
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: endpoint-observability-operator-fix
subjects:
- kind: ServiceAccount
  name: endpoint-observability-operator-sa
  namespace: open-cluster-management-addon-observability

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: endpoint-observability-operator-fix
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'