Bug 1886142
| Summary: | Operator managed PersistentVolumeClaims failing with 'cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Brandon Anderson <braander> | ||||
| Component: | apiserver-auth | Assignee: | Maru Newby <mnewby> | ||||
| Status: | CLOSED WORKSFORME | QA Contact: | pmali | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.3.z | CC: | aos-bugs, jsafrane, mfojtik | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.7.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-10-14 07:39:21 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Message "cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on" comes from some OCP auth. mechanism (SCC?), not storage. Clipped from the ClusterRole supplied in #c1: RBAC configuration for stateful sets > statefulsets.apps/finalizers [] [] [*] > statefulsets.apps [] [] [*] => Permission is granted to set finalizers for stateful sets. RBAC configuration for bookkeeper api group: > *.bookkeeper.pravega.io [] [] [*] => Permission is _not_ granted to set finalizers for resource types in the bookkeeper api group. - the yaml defining the ClusterRole [1] does not explicitly indicate the finalizer subresource for types like statefulset yet still grants permission to it - potential avenues for investigation: - Would permission be granted to finalizers if resources were explicitly named (rather than using a *.<api group> wildcard)? - Does the use of a wildcard for admissionregistration similarly preclude setting finalizers by members of the role? - Is finalizer security handled different between crds and built-in types? 1: https://github.com/pravega/bookkeeper-operator/blob/master/deploy/role.yaml#L48 I don't think the reported issue represents a bug. Rather, OpenShift is enforcing RBAC permissions and the service account used to run the pods created by the statefulset does not have permission to set the finalizers on the resource targeted by the volumeClaimTemplate's metadata.ownerReference. Manual testing confirms that ensuring that the update verb is allowed for <resource type>/finalizers should resolve the problem. Here are reports of similar issues that were resolved in this fashion: https://github.com/spotahome/redis-operator/issues/98 https://bugzilla.redhat.com/show_bug.cgi?id=1767806 https://access.redhat.com/solutions/5085891 https://github.com/operator-framework/operator-sdk/issues/1736#issuecomment-549433116 An SME provided the background for why OpenShift requires explicit permission for the /finalizer subresource in this instance. OpenShift enables a plugin that secures deletion via OwnerRef, and Kubernetes does not enable this plugin by default: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/gc/gc_admission.go#L35 I haven't yet found mention of this plugin or the implications of it being enabled in the openshift documentation, and am following up to see if I'm missing something or if a doc change is suggested. There does appear to be a gap in our documentation. I've added a jira story (`Document why and how OpenShift is 'Secure By Default'`) to ensure that differences in security configuration between OpenShift and upstream Kubernetes are documented. Hi, The cu has provided a new must-gather from their cluster. It is attached to case 02731014 and can be accessed via supportshell. Please let me know if you require any further data from the cu's cluster. (In reply to Brandon Anderson from comment #8) > Hi, > > The cu has provided a new must-gather from their cluster. It is attached to > case 02731014 and can be accessed via supportshell. Please let me know if > you require any further data from the cu's cluster. Please review the comment history. This bz has been closed. OpenShift is working as expected, and the resolution for the customer is to update their RBAC rules to explicitly allow access to the /finalizer endpoint for the resources in question. Feel free to each out to me on slack if these instructions are unclear. |
Created attachment 1719815 [details] pvc info Description of problem: Getting the following error when deploying statefulset, create Claim journal-bookkeeper-bookie-0 for Pod bookkeeper-bookie-0 in StatefulSet bookkeeper-bookie failed error: persistentvolumeclaims "journal-bookkeeper-bookie-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil> Generated from statefulset-controller 5 times in the last 2 minutes create Pod bookkeeper-bookie-0 in StatefulSet bookkeeper-bookie failed error: [failed to create PVC journal-bookkeeper-bookie-0: persistentvolumeclaims "journal-bookkeeper-bookie-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>, failed to create PVC ledger-bookkeeper-bookie-0: persistentvolumeclaims "ledger-bookkeeper-bookie-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>, failed to create PVC index-bookkeeper-bookie-0: persistentvolumeclaims "index-bookkeeper-bookie-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>] Finalizers are set in the clusterrole, [root@csah bookkeeper-operator]# oc describe clusterrole.rbac bkop-bookkeeper-operator Name: bkop-bookkeeper-operator Labels: app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=bookkeeper-operator app.kubernetes.io/version=0.1.2 helm.sh/chart=bookkeeper-operator-0.1.2 Annotations: meta.helm.sh/release-name: bkop meta.helm.sh/release-namespace: isilon PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- *.admissionregistration.k8s.io [] [] [*] bindings.apps/finalizers [] [] [*] daemonsets.apps [] [] [*] deployments.apps [] [] [*] replicasets.apps [] [] [*] statefulsets.apps/finalizers [] [] [*] statefulsets.apps [] [] [*] *.bookkeeper.pravega.io [] [] [*] poddisruptionbudgets.policy [] [] [*] configmaps/finalizers [] [] [get watch list create update delete] configmaps [] [] [get watch list create update delete] endpoints [] [] [get watch list create update delete] events [] [] [get watch list create update delete] nodes [] [] [get watch list create update delete] persistentvolumeclaims/finalizers [] [] [get watch list create update delete] persistentvolumeclaims [] [] [get watch list create update delete] pods/finalizers [] [] [get watch list create update delete] pods [] [] [get watch list create update delete] secrets [] [] [get watch list create update delete] services [] [] [get watch list create update delete] statefulsets/finalizers [] [] [get watch list create update delete] statefulsets [] [] [get watch list create update delete] Following is the volume claim template in the statefulset, volumeClaimTemplates: - metadata: creationTimestamp: null name: journal namespace: isilon ownerReferences: - apiVersion: bookkeeper.pravega.io/v1alpha1 blockOwnerDeletion: true controller: true kind: BookkeeperCluster name: bookkeeper uid: 671da881-4d24-4090-bf44-4a7691beec77 spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: standard volumeMode: Filesystem status: phase: Pending Version-Release number of selected component (if applicable): 4.3.x Actual results: PVC removal fails and kicks back this error: persistentvolumeclaims "journal-bookkeeper-bookie-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil> Expected results: PVC behaves as expected with this value set to true PVC info attached to case If any further info is required from customer or CU environment, please let me know and this can be retrieved.