Bug 1974648

Summary: GRC clusterrole does not have update permissions on other apigroups finalizers resources
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Ricardo Carrillo Cruz <ricarril>
Component: GRC & PolicyAssignee: Gus Parvin <gparvin>
Status: CLOSED CURRENTRELEASE QA Contact: Derek Ho <dho>
Severity: high Docs Contact: Mikela Dockery <mdockery>
Priority: high    
Version: rhacm-2.2CC: gparvin, juhsu, keyoung
Target Milestone: ---Flags: juhsu: rhacm-2.2.z-
Target Release: rhacm-2.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1974654 (view as bug list) Environment:
Last Closed: 2021-08-10 14:39:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1974654    

Description Ricardo Carrillo Cruz 2021-06-22 08:50:00 UTC
Description of the problem:

We are writing a Kubernetes operator that creates ACM secondary resources.
When the operator creates a primary resource that itself causes an ACM policy creation, the policy propagator pod logs shows this error:

<snip>

"level":"error","ts":1624022737.4564233,"logger":"policy-propagator","msg":"Failed to create replicated policy...","Policy-Namespace":"default","Policy-Name":"group1-batch-1-policy-deployment-policy","Namespace":"spoke1","Name":"default.group1-batch-1-policy-deployment-policy","error":"policies.policy.open-cluster-management.io \"default.group1-batch-1-policy-deployment-policy\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr.1/zapr.go:128\ngithub.com/open-cluster-management/governance-policy-propagator/pkg/controller/propagator.(*ReconcilePolicy).handleDecision\n\t/remote-source/app/pkg/controller/propagator/propagation.go:213\ngithub.com/open-cluster-management/governance-policy-propagator/pkg/controller/propagator.(*ReconcilePolicy).handleRootPolicy\n\t/remote-source/app/pkg/controller/propagator/propagation.go:81\ngithub.com/open-cluster-management/governance-policy-propagator/pkg/controller/propagator.(*ReconcilePolicy).Reconcile\n\t/remote-source/app/pkg/controller/propagator/policy_propagator_controller.go:153\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:90"}
{"level":"error","ts":1624022737.4568074,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"policy-propagator","request":"default/group1-batch-1-policy-deployment-policy","error":"policies.policy.open-cluster-management.io \"default.group1-batch-1-policy-deployment-policy\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery.3/pkg/util/wait/wait.go:90"}

</snip>

This is due to the grc clusterrole not having the update verb on group/finalizers resource of our operator.
Adding this to the clusterrole fixes the issue:

- apiGroups:
  - '*'
  resources:
  - '*/finalizers'
  verbs:
  - update

We kindly request to add these changes to the clusterrole specs, just so 3rd party operators can leverage ACM as an engine to perform changes on managed clusters. 

Release version:

2.2

Operator snapshot version:

OCP version:

4.8

Browser Info:

Steps to reproduce:
1. Clone https://github.com/rcarrillocruz/cluster-group-lcm/commit/d7b5c8ff942085fe9525f40f996da5c668b902c1
2. Run 'make deploy' with a KUBECONFIG targetting a hub cluster with ACM
3. Run oc apply -f group.yaml (group.yaml is located in the cloned repo)

Actual results:

Policy is not properly propagated to managed clusters and grc policy propagator pod logs shows error similar to the description above.

Expected results:

Policy propagator can update finalizers resources managed by 3rd party operators and policies can be propagated to managed clusters correctly.

Additional info:

Comment 1 Gus Parvin 2021-06-22 15:48:07 UTC
I was able to recreate this issue and certainly see the value of adding these roles.  I'll be reviewing this change with our secure engineering and rbac experts and will work on adding this so your operator will work as expected.  Thanks for trying out this scenario and reporting the bug!

Comment 2 Mike Ng 2021-06-24 14:01:49 UTC
G2Bsync 867070650 comment 
 gparvin Wed, 23 Jun 2021 18:37:58 UTC 
 G2Bsync Discussed with customer and found out this is not needed for ACM 2.2 z stream but delivery in to ACM 2.3 is ok.