Bug 1968122

Summary:	clusterdeployment fails because hiveadmission sc does not have correct permissions
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Chad Crum <ccrum>
Component:	Cluster Lifecycle	Assignee:	James Talton <jtalton>
Status:	CLOSED ERRATA	QA Contact:	Chad Crum <ccrum>
Severity:	low	Docs Contact:	Christopher Dawson <cdawson>
Priority:	unspecified
Version:	rhacm-2.3	CC:	aos-bugs, ccrum, jparrill, keyoung
Target Milestone:	---	Flags:	ming: rhacm-2.3+
Target Release:	rhacm-2.3
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-08-06 00:52:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chad Crum 2021-06-05 16:12:18 UTC

Description of problem:
Unable to create a ClusterDeployment CR because hiveadmission service account does not have proper permissions.

Version-Release number of selected component (if applicable):
2.3.0-DOWNSTREAM-2021-06-02-06-03-39
Hub cluster = OCP 4.8.0-fc.7
Env: ipv4 / libvirt

How reproducible:
Repeated several times

Steps to Reproduce:
1. Create ICSP to mirror production RHACM image urls to brew in hub
2. Create CatalogSource for 2.3.0-DOWNSTREAM-2021-06-02-06-03-39
3. Deploy ACM operator (Using "rhacm" as namespace)
4. Create MCH for ACM 
5. Create Assisted Service AgentServiceConfig in ACM namespace
6. Create CRDs for SNO deployment in a new empty namespace (ex sno-deployment [crd examples here: https://github.com/openshift/assisted-service/tree/master/docs/crds] 

Actual results:
ClusterDeployment CR will fail as it is unable to reach the hive admission api.


Expected results:
Able to create CD CR without issue.

Additional info:

My workaround was to give the hiveadmission sc cluster admin role, just to get it working.

Comment 2 Chad Crum 2021-06-07 12:47:19 UTC

I am trying to reproduce today with a fresh install.

Comment 3 Chad Crum 2021-06-07 23:44:28 UTC

I redeployed with fresh hub cluster today (still using 4.8.0-fc.7) and the same ACM DS build - I experienced different results:

- ClusterDeployment was able to be created (Using a new empty namespace outside of hive and acm ns's)
- Hiveadmission pods were running, which I assume is why the CD was able to be created
- I did still receive permission errors in the hiveadmission logs related to the service account:

E0607 23:33:10.717957       1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0607 23:33:12.624078       1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0607 23:33:27.592672       1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0607 23:33:31.429191       1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0607 23:34:07.804179       1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0607 23:34:19.112222       1 reflector.go:138] k8s.io/client-go.0+incompatible/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:hive:hiveadmission" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope

I'll leave this bz as Urgent for now because if this is an intermittent issue it definitely could be considered a blocker.

I will also retry to duplicate again.

Comment 6 Chad Crum 2021-07-13 13:05:30 UTC

I've tested deployment on 2.3.0-DOWNSTREAM-2021-07-12-03-45-43 and though the above messages still occur, they do not appear to have any impact. 

We've also been running regular QE CI against latest downstreams deploying various SNO cluster network configurations and do not see any impact.

Comment 8 errata-xmlrpc 2021-08-06 00:52:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Advanced Cluster Management for Kubernetes version 2.3), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3016