Description of problem: Cluster role system:controller:persistent-volume-binder need the create and delete secret permission for Azure file dynamic provision Version-Release number of selected component (if applicable): openshift v3.10.0-0.36.0 kubernetes v1.10.0+b81c8f8 How reproducible: Always Steps to Reproduce: 1.Create a azure file storage class 2.Create pvc which use this sc Actual results: Warning ProvisioningFailed <invalid> persistentvolume-controller Failed to provision volume with StorageClass "z1ne3m4x": Couldn't create secret secrets is forbidden: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot create secrets in the namespace "pmi2d": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot create secrets in project "pmi2d" Expected results: Should provision successfully According to https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-file We need to add the create permission of resource secret for clusterrole system:controller:persistent-volume-binder by default # oc get ClusterRole system:controller:persistent-volume-binder -o yaml - apiGroups: - "" attributeRestrictions: null resources: - secrets verbs: - get Additional info:
Since the namespace is not known in advance, it is not possible to create a default rule.
This needs to be documented as below: When using storage class to provision azure file shares that is to be used in a namespace, a corresponding rbac rule should be created to allow azure file provisioner to allow create secret in that namespace. assign to Traci.
*** Bug 1578583 has been marked as a duplicate of this bug. ***
(In reply to hchen from comment #2) > This needs to be documented as below: > > When using storage class to provision azure file shares that is to be used > in a namespace, a corresponding rbac rule should be created to allow azure > file provisioner to allow create secret in that namespace. > > assign to Traci. Where should this be documented? We have information on Azure File volumes here: https://docs.openshift.com/container-platform/3.9/install_config/persistent_storage/persistent_storage_azure_file.html
Adding vigoyal as the Docs Contact.
``` During provision, a secret is created for mounting credentials. If the cluster has enabled both RBAC and Controller Roles, add the create permission of resource secret for clusterrole system:controller:persistent-volume-binder. ``` I am a newbie to authorization: It seems that system:controller:persistent-volume-binder is a clusterrole, ie, not associated with any namespace. Would it be proper to just add `create` permission to `secrets` resource by default? Thanks.
> Where should this be documented? We have information on Azure File volumes > here: > > https://docs.openshift.com/container-platform/3.9/install_config/ > persistent_storage/persistent_storage_azure_file.html Helle Traci, I believe that hchen is asking for the way to do dynamic provisioning based on sc of azure file. The document above follows the load of creating a PV and using it in a PVC.
(In reply to hchen from comment #1) > Since the namespace is not known in advance, it is not possible to create a > default rule. @Huamin, The role system:controller:persistent-volume-binder is cluster role which used by azure file sc to create secrets in end user's project. And I agree with Hongkai's comment #6, it does not need to know the name space in advance, the user just need to create a pvc and use the sc, and then the system:controller:persistent-volume-binder will create the secret in the user's project. # oc get sc sc-hkl5e -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: 2018-05-21T08:10:09Z name: sc-hkl5e resourceVersion: "94194" selfLink: /apis/storage.k8s.io/v1/storageclasses/sc-hkl5e uid: 61295b07-5cce-11e8-9424-000d3a1ae476 provisioner: kubernetes.io/azure-file reclaimPolicy: Delete volumeBindingMode: Immediate # oc get pvc -o yaml apiVersion: v1 items: - apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-class: sc-hkl5e volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-file creationTimestamp: 2018-05-21T08:10:12Z finalizers: - kubernetes.io/pvc-protection name: azpvc namespace: hkl5e resourceVersion: "94215" selfLink: /api/v1/namespaces/hkl5e/persistentvolumeclaims/azpvc uid: 6296f848-5cce-11e8-9424-000d3a1ae476 spec: accessModes: - ReadWriteMany resources: requests: storage: 1Gi volumeMode: Filesystem volumeName: pvc-6296f848-5cce-11e8-9424-000d3a1ae476 status: accessModes: - ReadWriteMany capacity: storage: 1Gi phase: Bound kind: List metadata: resourceVersion: "" selfLink: "" # oc get pv pvc-6296f848-5cce-11e8-9424-000d3a1ae476 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: kubernetes.io/createdby: azure-file-dynamic-provisioner pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: kubernetes.io/azure-file creationTimestamp: 2018-05-21T08:10:14Z finalizers: - kubernetes.io/pv-protection name: pvc-6296f848-5cce-11e8-9424-000d3a1ae476 resourceVersion: "94212" selfLink: /api/v1/persistentvolumes/pvc-6296f848-5cce-11e8-9424-000d3a1ae476 uid: 64135c97-5cce-11e8-9424-000d3a1ae476 spec: accessModes: - ReadWriteMany azureFile: secretName: azure-storage-account-dsb24acb944ea411e883af0-secret secretNamespace: hkl5e shareName: kubernetes-dynamic-pvc-6296f848-5cce-11e8-9424-000d3a1ae476 capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: azpvc namespace: hkl5e resourceVersion: "94204" uid: 6296f848-5cce-11e8-9424-000d3a1ae476 persistentVolumeReclaimPolicy: Delete storageClassName: sc-hkl5e volumeMode: Filesystem status: phase: Bound # oc get secrets -n hkl5e NAME TYPE DATA AGE azure-storage-account-dsb24acb944ea411e883af0-secret Opaque 2 10m So I think we need to ask authorization to add the "create" and "delete"( which used by deleting secret after user delete pvc) to the Cluster role system:controller:persistent-volume-binder by default. Just like other dynamic provision, we do not need to do any authorization work before using azure file dynamic provision.
The Storage team will follow up on this bug.
As comment #2, below are the steps to make azure file dynamic provision work. 1. Admin need to create the role in user's project as below: $ cat azf-role.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: system:controller:persistent-volume-binder namespace: <user's project name> rules: - apiGroups: [""] resources: ["secrets"] verbs: ["create", "get", "delete"] 2. Admin need to create the role binding to the service account "persistent-volume-binder" in "kube-system" project $ cat azf-rolebind.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: system:controller:persistent-volume-binder namespace: <user's project> roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: system:controller:persistent-volume-binder subjects: - kind: ServiceAccount name: persistent-volume-binder namespace: kube-system 3. Admin add service account as admin to user's project oc policy add-role-to-user admin system:serviceaccount:kube-system:persistent-volume-binder -n <user's project> 4 Admin create storage class of azure file $ cat azfsc.yaml | oc create -f - kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: azfsc provisioner: kubernetes.io/azure-file mountOptions: - dir_mode=0777 - file_mode=0777 5. Then end user create a pvc which use this sc, azure file dynamic provision can work
Thank you Wenqi for the instructions. Traci, can these steps be documented? Thanks
(In reply to hchen from comment #11) > Thank you Wenqi for the instructions. > > > Traci, can these steps be documented? Thanks Yes, I just need to know where this should be documented. Is this new content or can this be added to existing content? Thanks.
I think we can add this to below doc? https://docs.openshift.com/container-platform/3.9/install_config/persistent_storage/dynamically_provisioning_pvs.html? Because this is related to dynamic provision and storage class, to add azure file in it is more appropriate. Thanks.
Work in progress: https://github.com/openshift/openshift-docs/pull/9991
(In reply to Traci Morrison from comment #14) > Work in progress: https://github.com/openshift/openshift-docs/pull/9991 Made changes as described: Added a section for the Azure file. This can be reviewed here: https://github.com/openshift/openshift-docs/pull/9991. Moving this bug to MODIFIED.
Merged: https://github.com/openshift/openshift-docs/pull/9991 CP to 3.10: https://github.com/openshift/openshift-docs/pull/10167
Published for 3.10: https://docs.openshift.com/container-platform/3.10/install_config/persistent_storage/dynamically_provisioning_pvs.html#azure-file-secret-permission