Bug 1575933 - [DOCS] Azure file: need add create and delete permission to secrets with cluster role persistent-volume-binder by default
Summary: [DOCS] Azure file: need add create and delete permission to secrets with clus...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.10.0
Assignee: Traci Morrison
QA Contact: Vikram Goyal
Vikram Goyal
URL:
Whiteboard:
: 1578583 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-08 10:32 UTC by Wenqi He
Modified: 2018-07-31 12:25 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-31 12:25:54 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Wenqi He 2018-05-08 10:32:19 UTC
Description of problem:
Cluster role system:controller:persistent-volume-binder need the create and delete secret permission for Azure file dynamic provision

Version-Release number of selected component (if applicable):
openshift v3.10.0-0.36.0
kubernetes v1.10.0+b81c8f8


How reproducible:
Always

Steps to Reproduce:
1.Create a azure file storage class
2.Create pvc which use this sc


Actual results:

 Warning ProvisioningFailed <invalid> persistentvolume-controller Failed to provision volume with StorageClass "z1ne3m4x": Couldn't create secret secrets is forbidden: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot create secrets in the namespace "pmi2d": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot create secrets in project "pmi2d"


Expected results:
Should provision successfully 
According to https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-file
We need to add the create permission of resource secret for clusterrole system:controller:persistent-volume-binder by default

# oc get ClusterRole system:controller:persistent-volume-binder -o yaml
- apiGroups:
  - ""
  attributeRestrictions: null
  resources:
  - secrets
  verbs:
  - get

Additional info:

Comment 1 hchen 2018-05-18 15:00:13 UTC
Since the namespace is not known in advance, it is not possible to create a default rule.

Comment 2 hchen 2018-05-18 15:04:09 UTC
This needs to be documented as below:

When using storage class to provision azure file shares that is to be used in a namespace, a corresponding rbac rule should be created to allow azure file provisioner to allow create secret in that namespace.

assign to Traci.

Comment 3 Hongkai Liu 2018-05-18 18:36:29 UTC
*** Bug 1578583 has been marked as a duplicate of this bug. ***

Comment 4 Traci Morrison 2018-05-18 19:16:41 UTC
(In reply to hchen from comment #2)
> This needs to be documented as below:
> 
> When using storage class to provision azure file shares that is to be used
> in a namespace, a corresponding rbac rule should be created to allow azure
> file provisioner to allow create secret in that namespace.
> 
> assign to Traci.

Where should this be documented? We have information on Azure File volumes here:

https://docs.openshift.com/container-platform/3.9/install_config/persistent_storage/persistent_storage_azure_file.html

Comment 5 Traci Morrison 2018-05-18 19:18:57 UTC
Adding vigoyal@redhat.com as the Docs Contact.

Comment 6 Hongkai Liu 2018-05-18 19:21:44 UTC
```
During provision, a secret is created for mounting credentials. If the cluster has enabled both RBAC and Controller Roles, add the create permission of resource secret for clusterrole system:controller:persistent-volume-binder.
```

I am a newbie to authorization:

It seems that system:controller:persistent-volume-binder is a clusterrole, ie, not associated with any namespace.
Would it be proper to just add `create` permission to `secrets` resource by default?

Thanks.

Comment 7 Hongkai Liu 2018-05-18 19:25:22 UTC
> Where should this be documented? We have information on Azure File volumes
> here:
> 
> https://docs.openshift.com/container-platform/3.9/install_config/
> persistent_storage/persistent_storage_azure_file.html

Helle Traci,

I believe that hchen is asking for the way to do dynamic provisioning based on sc of azure file.

The document above follows the load of creating a PV and using it in a PVC.

Comment 8 Wenqi He 2018-05-21 08:29:42 UTC
(In reply to hchen from comment #1)
> Since the namespace is not known in advance, it is not possible to create a
> default rule.

@Huamin, The role system:controller:persistent-volume-binder is cluster role which used by azure file sc to create secrets in end user's project. And I agree with Hongkai's comment #6, it does not need to know the name space in advance, the user just need to create a pvc and use the sc, and then the system:controller:persistent-volume-binder will create the secret in the user's project.

# oc get sc sc-hkl5e -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: 2018-05-21T08:10:09Z
  name: sc-hkl5e
  resourceVersion: "94194"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/sc-hkl5e
  uid: 61295b07-5cce-11e8-9424-000d3a1ae476
provisioner: kubernetes.io/azure-file
reclaimPolicy: Delete
volumeBindingMode: Immediate

# oc get pvc -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-class: sc-hkl5e
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-file
    creationTimestamp: 2018-05-21T08:10:12Z
    finalizers:
    - kubernetes.io/pvc-protection
    name: azpvc
    namespace: hkl5e
    resourceVersion: "94215"
    selfLink: /api/v1/namespaces/hkl5e/persistentvolumeclaims/azpvc
    uid: 6296f848-5cce-11e8-9424-000d3a1ae476
  spec:
    accessModes:
    - ReadWriteMany
    resources:
      requests:
        storage: 1Gi
    volumeMode: Filesystem
    volumeName: pvc-6296f848-5cce-11e8-9424-000d3a1ae476
  status:
    accessModes:
    - ReadWriteMany
    capacity:
      storage: 1Gi
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

# oc get pv pvc-6296f848-5cce-11e8-9424-000d3a1ae476 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: azure-file-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/azure-file
  creationTimestamp: 2018-05-21T08:10:14Z
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-6296f848-5cce-11e8-9424-000d3a1ae476
  resourceVersion: "94212"
  selfLink: /api/v1/persistentvolumes/pvc-6296f848-5cce-11e8-9424-000d3a1ae476
  uid: 64135c97-5cce-11e8-9424-000d3a1ae476
spec:
  accessModes:
  - ReadWriteMany
  azureFile:
    secretName: azure-storage-account-dsb24acb944ea411e883af0-secret
    secretNamespace: hkl5e
    shareName: kubernetes-dynamic-pvc-6296f848-5cce-11e8-9424-000d3a1ae476
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: azpvc
    namespace: hkl5e
    resourceVersion: "94204"
    uid: 6296f848-5cce-11e8-9424-000d3a1ae476
  persistentVolumeReclaimPolicy: Delete
  storageClassName: sc-hkl5e
  volumeMode: Filesystem
status:
  phase: Bound

# oc get secrets -n hkl5e
NAME                                                   TYPE                                  DATA      AGE
azure-storage-account-dsb24acb944ea411e883af0-secret   Opaque                                2         10m

So I think we need to ask authorization to add the "create" and "delete"( which used by deleting secret after user delete pvc) to the Cluster role system:controller:persistent-volume-binder by default. Just like other dynamic provision, we do not need to do any authorization work before using azure file dynamic provision.

Comment 9 Traci Morrison 2018-05-21 14:30:40 UTC
The Storage team will follow up on this bug.

Comment 10 Wenqi He 2018-05-31 08:42:42 UTC
As comment #2, below are the steps to make azure file dynamic provision work.

1. Admin need to create the role in user's project as below:
$ cat azf-role.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: system:controller:persistent-volume-binder
  namespace: <user's project name> 
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]

2. Admin need to create the role binding to the service account "persistent-volume-binder" in "kube-system" project

$ cat azf-rolebind.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: system:controller:persistent-volume-binder
  namespace: <user's project> 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: system:controller:persistent-volume-binder
subjects:
- kind: ServiceAccount
  name: persistent-volume-binder
namespace: kube-system

3. Admin add service account as admin to user's project
oc policy add-role-to-user admin system:serviceaccount:kube-system:persistent-volume-binder -n <user's project>

4 Admin create storage class of azure file

$ cat azfsc.yaml | oc create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azfsc
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777

5. Then end user create a pvc which use this sc, azure file dynamic provision can work

Comment 11 hchen 2018-05-31 12:09:37 UTC
Thank you Wenqi for the instructions.


Traci, can these steps be documented? Thanks

Comment 12 Traci Morrison 2018-06-01 17:47:38 UTC
(In reply to hchen from comment #11)
> Thank you Wenqi for the instructions.
> 
> 
> Traci, can these steps be documented? Thanks

Yes, I just need to know where this should be documented. Is this new content or can this be added to existing content? Thanks.

Comment 13 Wenqi He 2018-06-04 06:24:29 UTC
I think we can add this to below doc? https://docs.openshift.com/container-platform/3.9/install_config/persistent_storage/dynamically_provisioning_pvs.html?

Because this is related to dynamic provision and storage class, to add azure file in it is more appropriate. Thanks.

Comment 14 Traci Morrison 2018-06-11 17:30:15 UTC
Work in progress: https://github.com/openshift/openshift-docs/pull/9991

Comment 15 Traci Morrison 2018-06-11 17:37:26 UTC
(In reply to Traci Morrison from comment #14)
> Work in progress: https://github.com/openshift/openshift-docs/pull/9991

Made changes as described: Added a section for the Azure file. This can be reviewed here: https://github.com/openshift/openshift-docs/pull/9991. Moving this bug to MODIFIED.


Note You need to log in before you can comment on or make changes to this bug.