Bug 1839965
| Summary: | [azure-file] "Permission denied" when read/write to mounted directory on RHEL node. | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wei Duan <wduan> | |
| Component: | Installer | Assignee: | Russell Teague <rteague> | |
| Installer sub component: | openshift-ansible | QA Contact: | Gaoyun Pei <gpei> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | high | |||
| Priority: | unspecified | CC: | aos-bugs, jsafrane | |
| Version: | 4.5 | |||
| Target Milestone: | --- | |||
| Target Release: | 4.5.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
SELinux permissions were blocking read/write access to mounted volumes on the Azur platform. SELinux booleans are updated to match RHCOS 8.x to allow proper access.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1845830 1845836 1845837 (view as bug list) | Environment: | ||
| Last Closed: | 2020-07-13 17:41:29 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1845830 | |||
The cluster is 3 master(coreos) + 3 worker(RHEL), we tried to make pod assign to master node), mount/read/write works well. [wduan@MINT 01_general]$ oc get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME wduan0526-vt4fq-master-0 Ready master 4h22m v1.18.2 10.0.0.6 <none> Red Hat Enterprise Linux CoreOS 45.81.202005252026-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 wduan0526-vt4fq-master-1 Ready master 4h22m v1.18.2 10.0.0.8 <none> Red Hat Enterprise Linux CoreOS 45.81.202005252026-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 wduan0526-vt4fq-master-2 Ready master 4h23m v1.18.2 10.0.0.7 <none> Red Hat Enterprise Linux CoreOS 45.81.202005252026-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 wduan0526-vt4fq-rhel-0 Ready worker 3h14m v1.18.2 10.0.1.6 <none> Red Hat Enterprise Linux Server 7.8 (Maipo) 3.10.0-1127.8.2.el7.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el7 wduan0526-vt4fq-rhel-1 Ready worker 3h14m v1.18.2 10.0.1.7 <none> Red Hat Enterprise Linux Server 7.8 (Maipo) 3.10.0-1127.8.2.el7.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el7 wduan0526-vt4fq-rhel-2 Ready worker 3h14m v1.18.2 10.0.1.8 <none> Red Hat Enterprise Linux Server 7.8 (Maipo) 3.10.0-1127.8.2.el7.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el7 [wduan@MINT 01_general]$ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES dpod-4b4nr 1/1 Running 0 103s 10.129.2.28 wduan0526-vt4fq-rhel-0 <none> <none> dpod-7bdnj 1/1 Running 0 3m10s 10.129.0.56 wduan0526-vt4fq-master-2 <none> <none> dpod-c2jt6 1/1 Running 0 3m10s 10.130.0.37 wduan0526-vt4fq-master-1 <none> <none> dpod-cfv85 1/1 Running 0 3m10s 10.128.0.37 wduan0526-vt4fq-master-0 <none> <none> dpod-ghs2d 1/1 Running 0 2m 10.130.2.5 wduan0526-vt4fq-rhel-1 <none> <none> dpod-td2jx 1/1 Running 0 2m10s 10.131.2.6 wduan0526-vt4fq-rhel-2 <none> <none> [wduan@MINT 01_general]$ oc rsh dpod-cfv85 sh-4.4$ touch /mnt/storage/aaa sh-4.4$ ls /mnt/storage aaa test sh-4.4$ ls -lZ /mnt total 0 drwxrwxrwx. 2 1500 1500 system_u:object_r:cifs_t:s0 0 May 26 05:27 storage sh-4.4$ ps -Z LABEL PID TTY TIME CMD system_u:system_r:container_t:s0:c19,c24 16 pts/0 00:00:00 sh system_u:system_r:container_t:s0:c19,c24 28 pts/0 00:00:00 ps Reason is that master has SELinux boolean virt_use_samba "on", while the RHEL7.8 nodes have it "off". Simple fix: setsebool -P virt_use_samba 1 That needs to be added to documentation, however, I wasn't able to find any documentation that we actually support OCP with RHEL 7 nodes. Found it, https://docs.openshift.com/container-platform/4.4/machine_management/adding-rhel-compute.html I compared all filesystem related SELinux booleans between RHCOS 8.x and RHEL 7.8, these should be set on RHEL7.x via scaleup.yml to be on par with RHEL 8: virt_use_samba "on" container_use_cephfs "on" Verify this bug with openshift-ansible-4.5.0-202005271957.git.1.af03ff7.el7.noarch.rpm
During RHEL worker scale-up:
TASK [openshift_node : Setting sebool virt_use_samba] **************************
Friday 29 May 2020 19:17:06 +0800 (0:00:01.422) 0:05:25.936 ************
changed: [10.0.1.7] => {"changed": true, "name": "virt_use_samba", "persistent": true, "state": true}
changed: [10.0.1.6] => {"changed": true, "name": "virt_use_samba", "persistent": true, "state": true}
TASK [openshift_node : Setting sebool container_use_cephfs] ********************
Friday 29 May 2020 19:17:07 +0800 (0:00:01.151) 0:05:27.087 ************
changed: [10.0.1.7] => {"changed": true, "name": "container_use_cephfs", "persistent": true, "state": true}
changed: [10.0.1.6] => {"changed": true, "name": "container_use_cephfs", "persistent": true, "state": true}
Check the related SElinux booleans on the RHEL worker:
[root@gpei-455-9jxvj-rhel-0 cloud-user]# getsebool virt_use_samba
virt_use_samba --> on
[root@gpei-455-9jxvj-rhel-0 cloud-user]# getsebool container_use_cephfs
container_use_cephfs --> on
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |
Description of problem: When mount azure-file on RHEL(7.8) node, I try to read/write something but got "Permission denied". From the node, we can write something to the mounted directory. Version-Release number of selected component (if applicable): [wduan@MINT 01_general]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-05-25-232952 True False 3h10m Cluster version is 4.5.0-0.nightly-2020-05-25-232952 How reproducible: Always Steps to Reproduce: 1. Create rwx pvc with azure-file 2. Created damonset using pvc 3. Try to list forder and touch file in pod under mounted directory [wduan@MINT 01_general]$ oc get pod NAME READY STATUS RESTARTS AGE dpod-7v665 1/1 Running 0 62m dpod-hbkdr 1/1 Running 0 62m dpod-x2gsh 1/1 Running 0 62m [wduan@MINT 01_general]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc-rwx Bound pvc-1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 2Gi RWX azurefile-ocp 73m [wduan@MINT 01_general]$ oc rsh dpod-7v665 ls -lZ /mnt total 0 drwxrwxrwx. 2 1500 1500 system_u:object_r:cifs_t:s0 0 May 26 05:27 storage sh-4.4$ ps -Z LABEL PID TTY TIME CMD system_u:system_r:container_t:s0:c19,c24 57 pts/0 00:00:00 sh system_u:system_r:container_t:s0:c19,c24 63 pts/0 00:00:00 ps drwxrwxrwx. 2 1500 1500 system_u:object_r:cifs_t:s0 0 May 26 05:27 storage sh-4.4$ ls /mnt/storage/ ls: cannot open directory '/mnt/storage/': Permission denied sh-4.4$ touch /mnt/storage/testfile touch: cannot touch '/mnt/storage/testfile': Permission denied Actual results: mounted directory have no read/write access right Expected results: mounted directory should have read/write access right Master Log: Node Log (of failed PODs): sh-4.2# more /etc/redhat-release Red Hat Enterprise Linux Server release 7.8 (Maipo) PV Dump: [wduan@MINT 01_general]$ oc get pv pvc-1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: kubernetes.io/createdby: azure-file-dynamic-provisioner pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: kubernetes.io/azure-file creationTimestamp: "2020-05-26T05:27:23Z" finalizers: - kubernetes.io/pv-protection name: pvc-1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 resourceVersion: "71436" selfLink: /api/v1/persistentvolumes/pvc-1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 uid: 25d5a3de-da86-46f8-a45c-7a3f33ae627f spec: accessModes: - ReadWriteMany azureFile: secretName: azure-storage-account-fbcea863d20444b57822569-secret secretNamespace: wduan shareName: wduan0526-vt4fq-dynami-pvc-1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 capacity: storage: 2Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: mypvc-rwx namespace: wduan resourceVersion: "71427" uid: 1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 mountOptions: - uid=1500 - gid=1500 - mfsymlinks persistentVolumeReclaimPolicy: Delete storageClassName: azurefile-ocp volumeMode: Filesystem status: phase: Bound [wduan@MINT 01_general]$ PVC Dump: [wduan@MINT 01_general]$ oc get pvc mypvc-rwx -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-file volume.kubernetes.io/selected-node: wduan0526-vt4fq-rhel-2 creationTimestamp: "2020-05-26T05:27:20Z" finalizers: - kubernetes.io/pvc-protection name: mypvc-rwx namespace: wduan resourceVersion: "71438" selfLink: /api/v1/namespaces/wduan/persistentvolumeclaims/mypvc-rwx uid: 1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 spec: accessModes: - ReadWriteMany resources: requests: storage: 2Gi storageClassName: azurefile-ocp volumeMode: Filesystem volumeName: pvc-1c9c1dda-de21-4c1e-8f1f-e8d6cb253ba1 status: accessModes: - ReadWriteMany capacity: storage: 2Gi phase: Bound StorageClass Dump (if StorageClass used by PV/PVC): [wduan@MINT 01_general]$ oc get sc azurefile-ocp -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: "2020-05-26T04:26:01Z" name: azurefile-ocp resourceVersion: "51126" selfLink: /apis/storage.k8s.io/v1/storageclasses/azurefile-ocp uid: 342dfbb6-8aec-46d8-85f8-7ac07b0709dd mountOptions: - uid=1500 - gid=1500 - mfsymlinks parameters: skuName: Standard_LRS provisioner: kubernetes.io/azure-file reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer Additional info: