Bug 1837421 - [manila-csi-driver-operator] Cannot write data to volume when using deployment with RWX PVC.
Summary: [manila-csi-driver-operator] Cannot write data to volume when using deploymen...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Mike Fedosin
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks: 1848214 1848785
TreeView+ depends on / blocked
 
Reported: 2020-05-19 13:11 UTC by Wei Duan
Modified: 2020-10-27 16:00 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1848214 (view as bug list)
Environment:
Last Closed: 2020-10-27 16:00:21 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift csi-driver-manila-operator pull 40 0 None closed Bug 1837421: Set mounted share permissions to 777 2021-01-19 19:29:20 UTC
Github openshift csi-driver-nfs pull 28 0 None closed Bug 1837421: Fetch latest upstream patches 2021-01-19 19:29:20 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:00:50 UTC

Description Wei Duan 2020-05-19 13:11:26 UTC
Description of problem:
Cannot write data to volume when using the deployment with RWX PVC. (It is okay for pod with RWX PVC or  deployment with RWO PVC.)

Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-05-18-051243


How reproducible:
Always

Steps to Reproduce:
1. Manila csi driver is installed
2. Create deployment with RWX PVC (see yaml file in Additional info)
3. Write data to the mounted directory

Actual results:
[wduan@MINT 01_general]$ oc rsh mydeploy04-5677f4765c-n29ws
sh-4.4$ touch /mnt/local/testfile
touch: cannot touch '/mnt/local/testfile': Permission denied
sh-4.4$ ls -l /mnt/
total 1
drwxr-xr-x. 2 99 99 5 May 19 12:51 local
sh-4.4$ whoami 
1000600000


Expected results:
Write data should be successful.

Master Log:

Node Log (of failed PODs):
sh-4.4# cd /var/lib/kubelet/pods/0870998d-5d21-4616-afb5-d82e8e8f5723/volumes/kubernetes.io~csi/pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014/mount
sh-4.4# cd ..
sh-4.4# ls -LZ
    system_u:object_r:nfs_t:s0 mount  system_u:object_r:var_lib_t:s0 vol_data.json

PV Dump:
[wduan@MINT 01_general]$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS      REASON   AGE
pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014   1Gi        RWX            Delete           Bound    wduan/mydep-pvc04      csi-manila-ceph            163m

[wduan@MINT 01_general]$ oc get pv pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: manila.csi.openstack.org
  creationTimestamp: "2020-05-19T10:23:36Z"
  finalizers:
  - kubernetes.io/pv-protection
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-05-19T10:23:36Z"
  name: pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014
  resourceVersion: "199455"
  selfLink: /api/v1/persistentvolumes/pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014
  uid: 0816015c-f927-40be-bec6-89392c01083c
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: mydep-pvc04
    namespace: wduan
    resourceVersion: "199424"
    uid: 6385f8dc-10e7-41e2-9e7e-51adc96e7014
  csi:
    driver: manila.csi.openstack.org
    fsType: ext4
    nodePublishSecretRef:
      name: csi-manila-secrets
      namespace: manila-csi
    nodeStageSecretRef:
      name: csi-manila-secrets
      namespace: manila-csi
    volumeAttributes:
      cephfs-mounter: fuse
      shareAccessID: 868dab3b-ff9c-48ca-a855-be1f7673ed3b
      shareID: 0d37defe-7382-493b-987e-f64320599a79
      storage.kubernetes.io/csiProvisionerIdentity: 1589856779685-8081-manila.csi.openstack.org
    volumeHandle: 0d37defe-7382-493b-987e-f64320599a79
  persistentVolumeReclaimPolicy: Delete
  storageClassName: csi-manila-ceph
  volumeMode: Filesystem
status:
  phase: Bound


PVC Dump:
[wduan@MINT 01_general]$ oc get pvc
NAME          STATUS   VOLUME                                     CAPACITY   
mydep-pvc04   Bound    pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014   1Gi        RWX            csi-manila-ceph   161m

[wduan@MINT 01_general]$ oc get pvc mydep-pvc04 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org
  creationTimestamp: "2020-05-19T10:23:31Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: mydep-pvc04
  namespace: wduan
  resourceVersion: "199461"
  selfLink: /api/v1/namespaces/wduan/persistentvolumeclaims/mydep-pvc04
  uid: 6385f8dc-10e7-41e2-9e7e-51adc96e7014
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1G
  storageClassName: csi-manila-ceph
  volumeMode: Filesystem
  volumeName: pvc-6385f8dc-10e7-41e2-9e7e-51adc96e7014
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  phase: Bound


StorageClass Dump (if StorageClass used by PV/PVC):
[wduan@MINT 01_general]$ oc get sc csi-manila-ceph -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2020-05-19T02:23:52Z"
  name: csi-manila-ceph
  resourceVersion: "31546"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/csi-manila-ceph
  uid: 4b7aeb9d-b14a-418f-b426-d009ac21a15e
parameters:
  csi.storage.k8s.io/node-publish-secret-name: csi-manila-secrets
  csi.storage.k8s.io/node-publish-secret-namespace: manila-csi
  csi.storage.k8s.io/node-stage-secret-name: csi-manila-secrets
  csi.storage.k8s.io/node-stage-secret-namespace: manila-csi
  csi.storage.k8s.io/provisioner-secret-name: csi-manila-secrets
  csi.storage.k8s.io/provisioner-secret-namespace: manila-csi
  type: ceph
provisioner: manila.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: Immediate


Additional info:
deployment with PVC RWX
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mydeploy04
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-openshift
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
      - name: hello-openshift
        #image: aosqe/hello-openshift
        image: docker.io/aosqe/storage@sha256:a05b96d373be86f46e76817487027a7f5b8b5f87c0ac18a246b018df11529b40
        ports:
        - containerPort: 80
        volumeMounts:
        - name: local
          mountPath: /mnt/local
      volumes:
      - name: local
        persistentVolumeClaim:
          claimName: mydep-pvc04

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mydep-pvc04
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1G
  storageClassName: csi-manila-ceph

Comment 1 Mike Fedosin 2020-05-27 21:10:28 UTC
That was an issue in our testing cloud with their Ceph cluster. Now writing works well:

$ oc exec -ti new-nfs-share-pod bash
root@new-nfs-share-pod:/# mount | grep nfs4
172.16.32.1:/volumes/_nogroup/6241164b-4ab0-4358-ad50-1d3a3146cd23 on /var/lib/www type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.128.2.92,local_lock=none,addr=172.16.32.1)
root@new-nfs-share-pod:/# cd /var/lib/www/
root@new-nfs-share-pod:/var/lib/www# echo "Hello Manila!" > hello
root@new-nfs-share-pod:/var/lib/www# cat hello 
Hello Manila!

For this reason I close this bug.

Comment 2 Wei Duan 2020-06-02 09:10:29 UTC
Hi Mike,

I re-test it today(20200602), deployment with RWX still doesn't work.
There are 3 deployments using the same yaml file, and the difference are:
case 1. Manila with RWX: from the user and dir access mode, it could not write.
case 2. Cinder with RWO: ok
case 3. Manila with RWO: ok

Please see the following output, I also attach my yaml file:


[wduan@MINT 01_general]$ oc get pod,pvc
NAME                                      READY   STATUS    RESTARTS   AGE
pod/mydeploy01-5d87c8668-wgzqh            1/1     Running   0          87m
pod/mydeploy02-6f777c58c9-jmdfd           1/1     Running   0          48m
pod/mydeploy03-84898b8d66-s7gns           1/1     Running   0          6m18s

NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/mydep-pvc01   Bound    pvc-bf92b5fc-efd2-4b73-8277-1f8eb49346b0   3Gi        RWX            csi-manila-ceph   87m
persistentvolumeclaim/mydep-pvc02   Bound    pvc-24793ed6-6509-4575-ba62-dfbca9a46e37   3Gi        RWO            standard          48m
persistentvolumeclaim/mydep-pvc03   Bound    pvc-7c6fe557-c52e-4eaa-8b66-edd7d194abf1   3Gi        RWO            csi-manila-ceph   6m20s


Manila RWX:
[wduan@MINT 01_general]$ oc rsh mydeploy01-5d87c8668-wgzqh
sh-4.4$ mount | grep nfs
172.16.32.1:/volumes/_nogroup/08715905-5d01-4c57-b932-7903b21347c8 on /mnt/local type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.129.2.11,local_lock=none,addr=172.16.32.1)
sh-4.4$ whoami
1000590000
sh-4.4$ ls -l mnt/
total 1
drwxr-xr-x. 2 99 99 0 Jun  2 06:55 local
sh-4.4$ touch /mnt/local/test
touch: cannot touch '/mnt/local/test': Permission denied



Cinder RWO:
[wduan@MINT 01_general]$ oc rsh mydeploy02-6f777c58c9-jmdfd 
sh-4.4$ whoami 
1000590000
sh-4.4$ ls -l mnt/
total 4
drwxrwsr-x. 3 root 1000590000 4096 Jun  2 07:34 local
sh-4.4$ touch /mnt/local/test
sh-4.4$ exit

Manila RWO:
[wduan@MINT 01_general]$ oc rsh mydeploy03-84898b8d66-s7gns
sh-4.4$ whoami
1000590000
sh-4.4$ ls -l /mnt       
total 1
drwxrwsr-x. 2 99 1000590000 0 Jun  2 08:16 local
sh-4.4$ touch /mnt/local/test
sh-4.4$ ls /mnt/local/test
/mnt/local/test


Yaml file:
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mydeploy01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-openshift
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
      - name: hello-openshift
        image: docker.io/aosqe/storage@sha256:a05b96d373be86f46e76817487027a7f5b8b5f87c0ac18a246b018df11529b40
        ports:
        - containerPort: 80
        volumeMounts:
        - name: local
          mountPath: /mnt/local
      volumes:
      - name: local
        persistentVolumeClaim:
          claimName: mydep-pvc01

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mydep-pvc01
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 3G
  storageClassName: csi-manila-ceph

Comment 15 Wei Duan 2020-07-14 01:51:37 UTC
Verified pass on 4.6.0-0.nightly-2020-07-07-233934

[wduan@MINT manila]$ oc rsh dpod-6wmr5 
sh-4.4$ mount | grep nfs
172.16.32.1:/volumes/_nogroup/a27a8b70-de7f-4a72-bec1-5a7593c91749 on /mnt/storage type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.131.0.19,local_lock=none,addr=172.16.32.1)

sh-4.4$ whoami
1000590000

sh-4.4$ ls -l /mnt
total 1
drwxrwxrwx. 2 99 99 0 Jun 22 08:00 storage

sh-4.4$ cp /etc/hosts /mnt/storage 
sh-4.4$ touch /mnt/storage/test
sh-4.4$ ls /mnt/storage
hosts  test

Tested with deployment and daemonset:
[wduan@MINT manila]$ oc get all
NAME                              READY   STATUS    RESTARTS   AGE
pod/dpod-6wmr5                    1/1     Running   0          7m28s
pod/dpod-pxkst                    1/1     Running   0          7m28s
pod/dpod-x6fvm                    1/1     Running   0          7m27s
pod/mydeploy02-6f777c58c9-gfchh   1/1     Running   0          10m

NAME                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/dpod   3         3         3       3            3           <none>          7m31s

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mydeploy02   1/1     1            1           23m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/mydeploy02-6f777c58c9   1         1         1       23m
[wduan@MINT manila]$ oc get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mydep-pvc02   Bound    pvc-ab85e46a-4552-47c0-a421-bf8226b3d357   3Gi        RWX            csi-manila-ceph   24m
mypvc-rwx     Bound    pvc-4f378cab-14ae-465b-bbc4-d0670f46f0e9   2Gi        RWX            csi-manila-ceph   8m22s
[wduan@MINT manila]$

Comment 17 errata-xmlrpc 2020-10-27 16:00:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.