Bug 2050363

Description of the problem:
Attempting to deploy latest RHACM 2.5 with latest bundled Infrastructure Operator - Infrastructure operator ends up crashing and is unable to deploy assisted service or related pods. 

Release version:
ACM: 2.5.0-DOWNSTREAM-2022-02-03-07-04-00
MCE: /\ same
Assisted service image us git sha = af605b2633785473075384aa761f93d2e98ee337

OCP version:
4.10 latest (ipv4 connected bm ipi hub env)

Steps to reproduce:
1. Deploy ipv4 bm ipi ocp 4.10 hub cluster on virtual (or real bm) 
2. Deploy ACM operator
3. Deploy MCE / MCH
4. Create AgentServiceConfig for assisted service

Actual results:
Infrastructure Operator pod continuously crashes and assisted service pod does not start

Expected results:
Assisted service pod starts


Additional info:
Infrastructure operator pod log:

I0203 18:35:54.934652       1 request.go:668] Waited for 1.036220807s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/agent.open-cluster-management.io/v1?timeout=32s
{"level":"info","ts":1643913358.3936846,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1643913362.9049158,"logger":"setup","msg":"starting manager"}
I0203 18:36:02.905205       1 leaderelection.go:243] attempting to acquire leader lease rhacm/86f835c3.agent-install.openshift.io...
{"level":"info","ts":1643913362.905278,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0203 18:36:18.360069       1 leaderelection.go:253] successfully acquired lease rhacm/86f835c3.agent-install.openshift.io
{"level":"info","ts":1643913378.3604057,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3604739,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.360483,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.360489,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605177,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605235,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.360529,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605347,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605404,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605468,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605525,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.360558,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605642,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="}
{"level":"info","ts":1643913378.3605692,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting Controller","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig"}
E0203 18:36:18.373845       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:36:18.373937       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:36:19.304055       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:36:19.335660       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:36:21.183062       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:36:22.120936       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:36:26.484660       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:36:27.507747       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:36:35.780204       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:36:38.675420       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:36:54.045476       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:37:01.431860       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:37:29.263359       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:37:30.351781       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0203 18:38:13.236045       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
E0203 18:38:15.798449       1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
{"level":"error","ts":1643913498.3609416,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Could not wait for Cache to sync","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","error":"failed to wait for agentserviceconfig caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/remote-source/assisted-service/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:195\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/remote-source/assisted-service/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/remote-source/assisted-service/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/manager/internal.go:697"}
{"level":"error","ts":1643913498.3614812,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"leader election lost"}
{"level":"error","ts":1643913498.3615696,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"context canceled"}
{"level":"error","ts":1643913498.361579,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"context canceled"}
{"level":"error","ts":1643913498.3614466,"logger":"setup","msg":"problem running manager","error":"failed to wait for agentserviceconfig caches to sync: timed out waiting for cache to be synced","stacktrace":"main.main\n\t/remote-source/assisted-service/app/cmd/operator/main.go:164\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:255"}