Description of problem: Unable to create a ClusterDeployment CR because hiveadmission service account does not have proper permissions. Version-Release number of selected component (if applicable): 2.3.0-DOWNSTREAM-2021-06-02-06-03-39 Hub cluster = OCP 4.8.0-fc.7 Env: ipv4 / libvirt How reproducible: Repeated several times Steps to Reproduce: 1. Create ICSP to mirror production RHACM image urls to brew in hub 2. Create CatalogSource for 2.3.0-DOWNSTREAM-2021-06-02-06-03-39 3. Deploy ACM operator (Using "rhacm" as namespace) 4. Create MCH for ACM 5. Create Assisted Service AgentServiceConfig in ACM namespace 6. Create CRDs for SNO deployment in a new empty namespace (ex sno-deployment [crd examples here: https://github.com/openshift/assisted-service/tree/master/docs/crds] Actual results: ClusterDeployment CR will fail as it is unable to reach the hive admission api. Expected results: Able to create CD CR without issue. Additional info: My workaround was to give the hiveadmission sc cluster admin role, just to get it working.
I am trying to reproduce today with a fresh install.
I redeployed with fresh hub cluster today (still using 4.8.0-fc.7) and the same ACM DS build - I experienced different results: - ClusterDeployment was able to be created (Using a new empty namespace outside of hive and acm ns's) - Hiveadmission pods were running, which I assume is why the CD was able to be created - I did still receive permission errors in the hiveadmission logs related to the service account: E0607 23:33:10.717957 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope E0607 23:33:12.624078 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope E0607 23:33:27.592672 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope E0607 23:33:31.429191 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope E0607 23:34:07.804179 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope E0607 23:34:19.112222 1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope I'll leave this bz as Urgent for now because if this is an intermittent issue it definitely could be considered a blocker. I will also retry to duplicate again.
I've tested deployment on 2.3.0-DOWNSTREAM-2021-07-12-03-45-43 and though the above messages still occur, they do not appear to have any impact. We've also been running regular QE CI against latest downstreams deploying various SNO cluster network configurations and do not see any impact.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Advanced Cluster Management for Kubernetes version 2.3), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3016