Description of problem: Create a pod that has glusterfs volume mounted with ReadWriteOnce mode, the files of the gluster volume can be read from the pod, but can not be written to. Version-Release number of selected component (if applicable): openshift v3.0.1.0-338-g9dfce43 kubernetes v1.0.0 How reproducible: Always Steps to Reproduce: 1. Create gluster endpoints, PV and PVC, the accessMode of the PV is ReadWriteOnce. oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/gluster/endpoints.json oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/gluster/glusterfs.json oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/gluster/claim.json 2. Make sure PV and PVC have bound, then create the pod oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/gluster/pod.json 3. Go to the container of the pod, make sure it r/w accessible: Actual results: After step 3: # oc exec gluster -it -- bash bash-4.3$ ls /mnt/gluster/ hello index.html test bash-4.3$ touch /mnt/gluster/test1 touch: cannot touch '/mnt/gluster/test1': Permission denied bash-4.3$ ls -Zdl /mnt/gluster/ drwxr-xr-x. 3 system_u:object_r:fusefs_t:s0 root root 63 Jul 20 01:45 /mnt/gluster/ Expected results: Should be able to create the file test1 to the mount dir /mnt/gluster Additional info:
This problem also exits when writing mounted iscsi volume from pod(after setting selinux to permissive): bash-4.2$ ls /mnt/iscsi/ -ldZ drwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /mnt/iscsi/ bash-4.2$ ls /mnt/iscsi/ lost+found bash-4.2$ touch /mnt/iscsi/testfile touch: cannot touch '/mnt/iscsi/testfile': Permission denied
Huamin, can you give this a look?
Jianwei, this could be a selinux issue, see https://bugzilla.redhat.com/show_bug.cgi?id=1231936
@hchen, following bug 1231936, I did 'setsebool virt_sandbox_use_fusefs 1', the mount dir is read accessible, but write inaccessible Here is the SELinux context of the mount dir bash-4.3$ ls -Zd /mnt/gluster/ drwxr-xr-x. root root system_u:object_r:fusefs_t:s0 /mnt/gluster/ bash-4.3$ touch /mnt/gluster/flie touch: cannot touch 'flie': Permission denied
Jianwei, thanks for the upodate. I think you might want to try svirt_sandbox_file_t instead of virt_sandbox_use_fusefs. See if you can try "chcon -R -t svirt_sandbox_file_t /path_to_your_gluster_mount_on_your_host" and see any difference.
On node: [root@openshift-115 ~]# getsebool virt_sandbox_use_fusefs virt_sandbox_use_fusefs --> off On master: [root@openshift-114 tmp]# oc exec gluster -n jhou -it -- bash bash-4.2$ ls /mnt/gluster/ ls: cannot open directory /mnt/gluster/: Permission denied Turn virt_sandbox_use_fusefs on [root@openshift-115 ~]# setsebool virt_sandbox_use_fusefs 1 [root@openshift-115 ~]# getsebool virt_sandbox_use_fusefs virt_sandbox_use_fusefs --> on [root@openshift-114 tmp]# oc exec gluster -n jhou -it -- bash bash-4.2$ ls /mnt/gluster/ hello index.html test bash-4.2$ touch /mnt/gluster/t1 touch: cannot touch '/mnt/gluster/t1': Permission denied bash-4.2$ ls /mnt/gluster/ -Zd drwxr-xr-x. root root system_u:object_r:fusefs_t:s0 /mnt/gluster/ The result is same with comment 4, could read, couldn't write. chcon operation not supported bash-4.2$ chcon -R -t svirt_sandbox_file_t /mnt/gluster/ chcon: failed to change context of 'index.html' to 'system_u:object_r:svirt_sandbox_file_t:s0': Operation not supported chcon: failed to change context of 'hello' to 'system_u:object_r:svirt_sandbox_file_t:s0': Operation not supported chcon: failed to change context of 'test' to 'system_u:object_r:svirt_sandbox_file_t:s0': Operation not supported chcon: failed to change context of '/mnt/gluster/' to 'system_u:object_r:svirt_sandbox_file_t:s0': Operation not supported
Also tried svirt_sandbox_file_t, it seems on the system this boolean is not defined. [root@openshift-115 ~]# getsebool svirt_sandbox_file_t Error getting active value for svirt_sandbox_file_t
Jianwei, Did you run "chcon" inside container? That command should be run on host. Can you check your environment, get your glusterfs volume label, and run the following docker exec test? Here is my gluster volume label: # ls -Zd /var/lib/openshift/openshift.local.volumes/pods/b1de7110-524f-11e5-bf46-b8ca3a627d6c/volumes/kubernetes.io~glusterfs drwxr-x---. root root system_u:object_r:svirt_sandbox_file_t:s0 /var/lib/openshift/openshift.local.volumes/pods/b1de7110-524f-11e5-bf46-b8ca3a627d6c/volumes/kubernetes.io~glusterfs My environment is: # oc version oc v3.0.1.0-528-g8c2fe51 kubernetes v1.0.0 # getenforce Permissive # docker inspect 71f90f66890e |grep gluster "/usr/share/nginx/html/test": "/var/lib/openshift/openshift.local.volumes/pods/b1de7110-524f-11e5-bf46-b8ca3a627d6c/volumes/kubernetes.io~glusterfs/gluster-default-volume", # docker exec -ti 71f90f66890e touch /usr/share/nginx/html/test/glusterfs/foobar # docker exec -ti 71f90f66890e ls -l /usr/share/nginx/html/test/glusterfs/foobar -rw-r--r--. 1 root root 0 Sep 24 09:59 /usr/share/nginx/html/test/glusterfs/foobar
I did "chcon" on the from the node where the container is hosted, but the operation is not allowed [root@openshift-117 ~]# chcon -R -t svirt_sandbox_file_t /var/lib/openshift/openshift.local.volumes/pods/acfbc766-6d62-11e5-97e3-fa163e53da5a/volumes/kubernetes.io~glusterfs/gluster chcon: failed to change context of ‘index.html’ to ‘system_u:object_r:svirt_sandbox_file_t:s0’: Operation not supported chcon: failed to change context of ‘hello’ to ‘system_u:object_r:svirt_sandbox_file_t:s0’: Operation not supported chcon: failed to change context of ‘test’ to ‘system_u:object_r:svirt_sandbox_file_t:s0’: Operation not supported chcon: failed to change context of ‘/var/lib/openshift/openshift.local.volumes/pods/acfbc766-6d62-11e5-97e3-fa163e53da5a/volumes/kubernetes.io~glusterfs/gluster’ to ‘system_u:object_r:svirt_sandbox_file_t:s0’: Operation not supported
Jianwei, thanks for the update. Your "chcon" went one level deeper. You have to apply the label to /var/lib/openshift/openshift.local.volumes/pods/acfbc766-6d62-11e5-97e3-fa163e53da5a/volumes/kubernetes.io~glusterfs/, rather than /var/lib/openshift/openshift.local.volumes/pods/acfbc766-6d62-11e5-97e3-fa163e53da5a/volumes/kubernetes.io~glusterfs/gluster. For more information, please check "Mounting External Volumes" from docker-run manpage.
We don't pass security label during mount. There are upstream efforts to do this. For now, please make your pod privileged to write to gluster volume.
I don't think you would be able to do a chcon on the client side for GlusterFS try running the above chcon on the server where Gluster is being served from
Thank you all for the updates, I have verified that with privileged pod, I'm able to r/w to the mount dir of glusterfs with selinux enforcing. Inside container: The security context is [root@gluster ~]# ls -lZd /mnt/gluster/ drwxr-xr-x. root root system_u:object_r:fusefs_t:s0 /mnt/gluster/ From node: [root@openshift-114 ~]# mount|grep gluster 10.66.79.108:testvol on /var/lib/origin/openshift.local.volumes/pods/743092a5-796e-11e5-9ca2-fa163ee4ad04/volumes/kubernetes.io~glusterfs/gluster1 type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [root@openshift-114 ~]# ls -lZd /var/lib/origin/openshift.local.volumes/pods/743092a5-796e-11e5-9ca2-fa163ee4ad04/volumes/kubernetes.io~glusterfs/ drwxr-x---. root root system_u:object_r:svirt_sandbox_file_t:s0 /var/lib/origin/openshift.local.volumes/pods/743092a5-796e-11e5-9ca2-fa163ee4ad04/volumes/kubernetes.io~glusterfs/ [root@openshift-114 ~]# ls -lZd /var/lib/origin/openshift.local.volumes/pods/743092a5-796e-11e5-9ca2-fa163ee4ad04/volumes/kubernetes.io~glusterfs/gluster1/ drwxr-xr-x. root root system_u:object_r:fusefs_t:s0 /var/lib/origin/openshift.local.volumes/pods/743092a5-796e-11e5-9ca2-fa163ee4ad04/volumes/kubernetes.io~glusterfs/gluster1/
This bug can be verified according to comment 14
This fix is available in OpenShift Enterprise 3.1.