Bug 1903062 - [Cinder csi driver] Deployment mounted volume have no write access
Summary: [Cinder csi driver] Deployment mounted volume have no write access
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Matthew Booth
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-01 09:00 UTC by Wei Duan
Modified: 2021-02-24 15:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:36:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openstack-cinder-csi-driver-operator pull 19 0 None closed Bug 1903062: Make default fsType explicit in csi-provisioner 2021-01-21 16:17:51 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:37:05 UTC

Description Wei Duan 2020-12-01 09:00:20 UTC
Description of problem:
When creating deployment with PVC using cinder csi driver, the mounted volume have no write access, looks like the FsGroup is not correctly setted.

$ oc rsh mydeploy03-55f486b9f-cspkj
sh-4.4$ id
uid=1000640000(1000640000) gid=0(root) groups=0(root),1000640000

sh-4.4$ ls -l /mnt
total 4
drwxr-xr-x. 3 root root 4096 Dec  1 07:15 local

sh-4.4$ ls -ldZ /mnt/local
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c20,c25 4096 Dec  1 07:15 /mnt/local

sh-4.4$ echo  "Hello csi cinder" > /mnt/local/hello
sh: /mnt/local/hello: Permission denied

The same scenario works well with cinder in-tree plugin.
The simple pod works well inder csi plugin. 


Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2020-11-29-133728

Steps to Reproduce:
1. Install OSP cluster and cinder csi driver is installed. 
2. Create PVC with cinder csi driver and create deployment to consume this PVC, mounted volume is /mnt/local
3. Write something into mounted volume
4. Compared with In-tree plugin


Actual results:
Mounted volume have no the write access when using cinder csi driver.
-------------------------------------
CSI driver case:

$ oc create -f dep.yaml
deployment.apps/mydeploy03 created
persistentvolumeclaim/mydep-pvc03 created

$ oc get pod
NAME                         READY   STATUS    RESTARTS   AGE
mydeploy03-55f486b9f-cspkj   1/1     Running   0          16s

$ oc get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mydep-pvc03   Bound    pvc-f3b8cebc-f143-48b9-a7e0-b1f5077ac39b   1Gi        RWO            standard-csi   33s

$ oc rsh mydeploy03-55f486b9f-cspkj
sh-4.4$ id
uid=1000640000(1000640000) gid=0(root) groups=0(root),1000640000

sh-4.4$ ls -l /mnt
total 4
drwxr-xr-x. 3 root root 4096 Dec  1 07:15 local

sh-4.4$ ls -ldZ /mnt/local
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c20,c25 4096 Dec  1 07:15 /mnt/local

sh-4.4$ echo  "Hello csi cinder" > /mnt/local/hello
sh: /mnt/local/hello: Permission denied

-------------------------------------
In-tree plugin

$ oc create -f dep.yaml
deployment.apps/mydeploy04 created
persistentvolumeclaim/mydep-pvc04 created

$ oc get pod
NAME                          READY   STATUS              RESTARTS   AGE
mydeploy04-7f558d7578-t859d   1/1     Running             0          22s

$ oc get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mydep-pvc04   Bound    pvc-d9f4d6df-cb4d-494f-8a2b-8eeec8006f97   1Gi        RWO            standard       28s

$ oc rsh mydeploy04-7f558d7578-t859d
sh-4.4$ id
uid=1000650000(1000650000) gid=0(root) groups=0(root),1000650000

sh-4.4$ cd /mnt
sh-4.4$ ls -l
total 4
drwxrwsr-x. 3 root 1000650000 4096 Dec  1 07:51 local
sh-4.4$ ls -ldZ /mnt/local

drwxrwsr-x. 3 root 1000650000 system_u:object_r:container_file_t:s0:c0,c26 4096 Dec  1 07:53 /mnt/local

sh-4.4$ echo  "Hello in-tree cinder" > /mnt/local/hello
sh-4.4$ more /mnt/local/hello
Hello in-tree cinder

-------------------------------------
Something differet for mount info on node

/dev/vdc on /var/lib/kubelet/pods/efa46102-e2f4-4478-9568-195f579d3675/volumes/kubernetes.io~cinder/pvc-d9f4d6df-cb4d-494f-8a2b-8eeec8006f97 type ext4 (rw,relatime,seclabel)
/dev/vdc on /var/lib/kubelet/pods/efa46102-e2f4-4478-9568-195f579d3675/volumes/kubernetes.io~cinder/pvc-d9f4d6df-cb4d-494f-8a2b-8eeec8006f97 type ext4 (rw,relatime,seclabel)

/dev/vdd on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f3b8cebc-f143-48b9-a7e0-b1f5077ac39b/globalmount type ext4 (rw,relatime,seclabel)
/dev/vdd on /var/lib/kubelet/pods/4e63c8f8-a821-405e-8b81-6db9262294da/volumes/kubernetes.io~csi/pvc-f3b8cebc-f143-48b9-a7e0-b1f5077ac39b/mount type ext4 (rw,relatime,seclabel)
/dev/vdd on /var/lib/kubelet/pods/4e63c8f8-a821-405e-8b81-6db9262294da/volumes/kubernetes.io~csi/pvc-f3b8cebc-f143-48b9-a7e0-b1f5077ac39b/mount type ext4 (rw,relatime,seclabel)
-------------------------------------


Expected results:
Mounted volume should have the write access when using cinder csi driver.


Additional info:
yaml file used:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mydeploy03
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-openshift
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
      - name: hello-openshift
        image: docker.io/aosqe/storage@sha256:a05b96d373be86f46e76817487027a7f5b8b5f87c0ac18a246b018df11529b40
        ports:
        - containerPort: 80
        volumeMounts:
        - name: local
          mountPath: /mnt/local
      volumes:
      - name: local
        persistentVolumeClaim:
          claimName: mydep-pvc03

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mydep-pvc03
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard-csi

Comment 1 Qin Ping 2020-12-02 09:57:50 UTC
The PV does not include fsType field, such as:
csi:
      driver: cinder.csi.openstack.org
      volumeAttributes:
        storage.kubernetes.io/csiProvisionerIdentity: 1606875731732-8081-cinder.csi.openstack.org
      volumeHandle: e4c7dc47-0fba-4ed5-9d8e-d1b9fd152b72

If creating a new storageclass with the following parameter:
parameters:
  csi.storage.k8s.io/fstype: ext4

Then the PV has the fsType field:
driver: cinder.csi.openstack.org
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1606875731732-8081-cinder.csi.openstack.org
    volumeHandle: 8fe26576-cbbf-41ce-93b9-c7ca776a8d4e

And the volume has write access right:
$ oc rsh deployment-4-85fd89cc87-hljgx 
sh-4.4# ls -ldZ /mnt/storage/
drwxrwsr-x. 2 root 1000610001 system_u:object_r:container_file_t:s0:c5,c25 6 Dec  2 08:08 /mnt/storage/

Comment 2 Wei Duan 2020-12-07 02:52:53 UTC
Just add one more scenario for single pod test:

Last test for a single pod in description session, I used the administrator user and the test passed. (Data can be written into pod)
But when using a common user, there is no write access neither. The user/group for mounted dir is root/root and docker run as a specific user/group as deployment case.

Comment 3 Martin André 2020-12-10 09:24:13 UTC
Again, more yaml tweaking.

Comment 4 Matthew Booth 2021-01-12 11:42:41 UTC
Reproduced on 4.7.0-0.nightly-2021-01-10-070949

Comment 5 Matthew Booth 2021-01-12 11:59:19 UTC
Created and used the following storageclass, which 'fixes' the problem:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-csi-fstype
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  fsType: ext4

Comment 6 Matthew Booth 2021-01-12 15:45:38 UTC
IIUC, the underlying cause of this issue is intentional behaviour by kubernetes: it doesn't set fsGroup during attach unless the volume meets a number of preconditions, one of which is an explicit fstype. This means that the implicit ext4 volume doesn't have its fsgroup set, while the explit ext4 volume does.

Apparently this may be addressed in the future more completely by: https://kubernetes-csi.github.io/docs/support-fsgroup.html

For now, we can fix this by making csi-provisioner add an explicit fstype when it is missing by adding '--default-fstype=ext4' to its args in openstack-cinder-csi-driver-controller.

Patch to follow.

Comment 9 Wei Duan 2021-01-22 06:15:22 UTC
Verified pass on upgrade from 4.6.9 to 4.7.0-0.nightly-2021-01-21-172657.

Deployment: 
$ oc create -f 04_deployment_pvc_sc.yaml
deployment.apps/mydeploy03 created
persistentvolumeclaim/mydep-pvc03 created
$ oc get pod
NAME                          READY   STATUS    RESTARTS   AGE
mydeploy03-55bfb56b4d-lkknd   1/1     Running   0          4m3s
$ oc rsh mydeploy03-55bfb56b4d-lkknd
sh-4.4# cd /mnt/local/
sh-4.4# touch a
sh-4.4# echo  "Hello csi cinder" > /mnt/local/hello
sh-4.4# echo /mnt/local/hello
/mnt/local/hello

pod with non-admin user:
$ oc rsh mypod
sh-4.4$ id
uid=1000630000(1000630000) gid=0(root) groups=0(root),1000630000
sh-4.4$ ls -ld /mnt/local/
drwxrwsr-x. 3 root 1000630000 4096 Jan 22 06:12 /mnt/local/
sh-4.4$ touch /mnt/local/test
sh-4.4$

Comment 12 errata-xmlrpc 2021-02-24 15:36:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.