Description of the problem: Attempting to deploy latest RHACM 2.5 with latest bundled Infrastructure Operator - Infrastructure operator ends up crashing and is unable to deploy assisted service or related pods. Release version: ACM: 2.5.0-DOWNSTREAM-2022-02-03-07-04-00 MCE: /\ same Assisted service image us git sha = af605b2633785473075384aa761f93d2e98ee337 OCP version: 4.10 latest (ipv4 connected bm ipi hub env) Steps to reproduce: 1. Deploy ipv4 bm ipi ocp 4.10 hub cluster on virtual (or real bm) 2. Deploy ACM operator 3. Deploy MCE / MCH 4. Create AgentServiceConfig for assisted service Actual results: Infrastructure Operator pod continuously crashes and assisted service pod does not start Expected results: Assisted service pod starts Additional info: Infrastructure operator pod log: I0203 18:35:54.934652 1 request.go:668] Waited for 1.036220807s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/agent.open-cluster-management.io/v1?timeout=32s {"level":"info","ts":1643913358.3936846,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1643913362.9049158,"logger":"setup","msg":"starting manager"} I0203 18:36:02.905205 1 leaderelection.go:243] attempting to acquire leader lease rhacm/86f835c3.agent-install.openshift.io... {"level":"info","ts":1643913362.905278,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} I0203 18:36:18.360069 1 leaderelection.go:253] successfully acquired lease rhacm/86f835c3.agent-install.openshift.io {"level":"info","ts":1643913378.3604057,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3604739,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.360483,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.360489,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605177,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605235,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.360529,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605347,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605404,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605468,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605525,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.360558,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605642,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting EventSource","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","source":"kind source: /, Kind="} {"level":"info","ts":1643913378.3605692,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Starting Controller","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig"} E0203 18:36:18.373845 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:36:18.373937 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:36:19.304055 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:36:19.335660 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:36:21.183062 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:36:22.120936 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:36:26.484660 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:36:27.507747 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:36:35.780204 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:36:38.675420 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:36:54.045476 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:37:01.431860 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:37:29.263359 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:37:30.351781 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope E0203 18:38:13.236045 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope E0203 18:38:15.798449 1 reflector.go:138] pkg/mod/k8s.io/client-go.1/tools/cache/reflector.go:167: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:rhacm:assisted-service" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope {"level":"error","ts":1643913498.3609416,"logger":"controller-runtime.manager.controller.agentserviceconfig","msg":"Could not wait for Cache to sync","reconciler group":"agent-install.openshift.io","reconciler kind":"AgentServiceConfig","error":"failed to wait for agentserviceconfig caches to sync: timed out waiting for cache to be synced","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2\n\t/remote-source/assisted-service/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:195\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start\n\t/remote-source/assisted-service/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:221\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/remote-source/assisted-service/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/manager/internal.go:697"} {"level":"error","ts":1643913498.3614812,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"leader election lost"} {"level":"error","ts":1643913498.3615696,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"context canceled"} {"level":"error","ts":1643913498.361579,"logger":"controller-runtime.manager","msg":"error received after stop sequence was engaged","error":"context canceled"} {"level":"error","ts":1643913498.3614466,"logger":"setup","msg":"problem running manager","error":"failed to wait for agentserviceconfig caches to sync: timed out waiting for cache to be synced","stacktrace":"main.main\n\t/remote-source/assisted-service/app/cmd/operator/main.go:164\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:255"}
*** This bug has been marked as a duplicate of bug 2046554 ***