Bug 1252627

Summary: Cannot set selinux context on files in on a glusterfs mount
Product: [Community] GlusterFS Reporter: Bob Arendt <rda>
Component: unclassifiedAssignee: bugs <bugs>
Status: CLOSED DEFERRED QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.3CC: bugs, howey.vernon, justin, mselvaga, ndevos, rcyriac, sankarshan, sebastian.gumprich
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1256635 (view as bug list) Environment:
Last Closed: 2016-06-30 10:51:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1318100    
Bug Blocks: 1256635, 1563506    

Description Bob Arendt 2015-08-11 23:12:50 UTC
Description of problem:
After creating a gluster volume on top of xfs partitions, and mounting that volume, I'm unable to change the security context of files on the mounted filesystem.

Version-Release number of selected component:  Tested with both

glusterfs-3.6.4-1.el6.x86_64
glusterfs-api-3.6.4-1.el6.x86_64
glusterfs-cli-3.6.4-1.el6.x86_64
glusterfs-debuginfo-3.6.4-1.el6.x86_64
glusterfs-extra-xlators-3.6.4-1.el6.x86_64
glusterfs-fuse-3.6.4-1.el6.x86_64
glusterfs-libs-3.6.4-1.el6.x86_64
glusterfs-server-3.6.4-1.el6.x86_64

glusterfs-3.7.3-1.el6.x86_64
glusterfs-api-3.7.3-1.el6.x86_64
glusterfs-cli-3.7.3-1.el6.x86_64
glusterfs-client-xlators-3.7.3-1.el6.x86_64
glusterfs-debuginfo-3.7.3-1.el6.x86_64
glusterfs-fuse-3.7.3-1.el6.x86_64
glusterfs-libs-3.7.3-1.el6.x86_64
glusterfs-server-3.7.3-1.el6.x86_64


How reproducible:  Always


Steps to Reproduce:  (for 3.7.3)
Built the latest RPM's from source from:
http://dl.fedoraproject.org/pub/epel/6/SRPMS/userspace-rcu-0.7.7-1.el6.src.rpm
http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/epel-6.6/SRPMS/glusterfs-3.7.3-1.el6.src.rpm

rpm -ivh glusterfs-3.7.3-1.el6.src.rpm
cd rpmbuild
rpmbuild -ba SPECS/glusterfs.spec |& tee log
  (put the rpm's in a private repository)

Create two test VM's, "ga" and "gb" using rhel66.  Each VM has partitions:
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda8        6558440 2802892   3415736  46% /
tmpfs             510028       0    510028   0% /dev/shm
/dev/vda3         200000   10400    189600   6% /b1
/dev/vda5         200000   10400    189600   6% /b2
/dev/vda6         200000   10400    189600   6% /b3
/dev/vda7         200000   10400    189600   6% /b4
/dev/vda1         243823   28113    202910  13% /boot

/dev/vda1: UUID="d3642293-57b1-4988-ac4f-85b0635e64c6" TYPE="ext4" 
/dev/vda2: UUID="f08a07bf-9222-45c7-9fd1-4d33207a8b86" TYPE="swap" 
/dev/vda3: UUID="ae1d0314-c22d-401e-a1b9-cf30fa4c6542" TYPE="xfs" 
/dev/vda5: UUID="5d32686c-5f00-4a00-ac06-539d2b85110d" TYPE="xfs" 
/dev/vda6: UUID="97e56854-76e6-4484-9afb-c5f56907df6e" TYPE="xfs" 
/dev/vda7: UUID="9e4d2bd4-de96-43f0-9aa3-3769cb22d508" TYPE="xfs" 
/dev/vda8: UUID="0be135b1-9bd5-49f5-8b21-8f43367a825b" TYPE="ext4" 

For the test we're only using the /b1 partitions on each host as our test bricks.

yum install glusterfs glusterfs-api glusterfs-cli glusterfs-client-xlators glusterfs-debuginfo glusterfs-fuse glusterfs-libs glusterfs-server

Create the a replicated volume:
chkconfig glusterd on
service glusterd start
gluster peer probe ga
gluster peer probe gb

gluster volume create gvol replica 2 transport tcp ga:/b1 gb:/b1 force
gluster volume start  gvol
gluster volume set    gvol auth.allow ga,gb
gluster volume set    gvol nfs.disable on

gluster volume info
    Volume Name: gvol
    Type: Replicate
    Volume ID: 4eeb493c-ed5f-4c3b-8945-4d14848a95d5
    Status: Started
    Number of Bricks: 1 x 2 = 2
    Transport-type: tcp
    Bricks:
    Brick1: ga:/b1
    Brick2: gb:/b1
    Options Reconfigured:
    nfs.disable: on
    auth.allow: ga,gb
    performance.readdir-ahead: on

On each host, mount the volume
mkdir /data
mount -t glusterfs -o selinux localhost:/gvol /data

Check that the --selinux switch is asserted ... 
# ps -eo args |grep glust
/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
/usr/sbin/glusterfsd -s gb --volfile-id gvol.gb.b1 -p /var/lib/glusterd/vols/gvol/run/gb-b1.pid -S /var/run/gluster/8dd23446126b2065164fdba21397998f.socket --brick-name /b1 -l /var/log/glusterfs/bricks/b1.log --xlator-option *-posix.glusterd-uuid=72fb7173-c1d3-4edd-b7e6-e7633e33358e --brick-port 49152 --xlator-option gvol-server.listen-port=49152
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a8c70c7c13620b79a8b5d26757294453.socket --xlator-option *replicate*.node-uuid=72fb7173-c1d3-4edd-b7e6-e7633e33358e
/usr/sbin/glusterfs --selinux --volfile-server=localhost --volfile-id=/gvol /data


Make some directories and files:

mkdir -p /data/a/b/c
echo test file > /data/a/b/myfile

Now for the test ...

Actual results:
[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:fusefs_t:s0    /data/a/b/myfile

[root@ga ~]# chcon -t tftpdir_rw_t /data/a/b/myfile
chcon: failed to change context of `/data/a/b/myfile' to `system_u:object_r:tftpdir_rw_t:s0': Operation not supported

[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:fusefs_t:s0    /data/a/b/myfile


Expected results:
[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:fusefs_t:s0    /data/a/b/myfile

[root@ga ~]# chcon -t tftpdir_rw_t /data/a/b/myfile

[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:tftpdir_rw_t:s0    /data/a/b/myfile


Additional info:
Works on other file systems:
[root@ga ~]# touch /tmp/x
[root@ga ~]# ls -Z /tmp/x
-rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 /tmp/x
[root@ga ~]# chcon -t tftpdir_rw_t /tmp/x
[root@ga ~]# ls -Z /tmp/x
-rw-r--r--. root root unconfined_u:object_r:tftpdir_rw_t:s0 /tmp/x

Both hosts have selinux enabled in permissive mode

The mount has selinux capability enabled.  Is there anything on the server side that needs to be configured to enable selinux capability?

Comment 1 Bob Arendt 2015-08-27 00:11:08 UTC
The man pages for glusterd, glusterfs, and glusterfsd processes indicate that they take a --selinux flag.  I tried applying this by hacking the glusterd code to add it - without success.  I tried:

/etc/sysconfig/glusterd add
   GLUSTERD_OPTIONS='--selinux'

Patch the glusterfs-3.7.3 build:
--- SPECS/glusterfs.spec_orig   2015-08-25 00:38:13.610000109 +0000
+++ SPECS/glusterfs.spec        2015-08-26 00:26:26.793000286 +0000
@@ -165,7 +165,7 @@
 %if ( 0%{_for_fedora_koji_builds} )
 Name:             glusterfs
 Version:          3.7.3
-Release:          1%{?prereltag:.%{prereltag}}%{?dist}
+Release:          4%{?prereltag:.%{prereltag}}%{?dist}rda
 Vendor:           Fedora Project
 %else
 Name:             @PACKAGE_NAME@
@@ -187,6 +187,8 @@
 Source0:          @PACKAGE_NAME@-@PACKAGE_VERSION@.tar.gz
 %endif
 
+Patch0:           glusterd-3.7.3-selinux.patch
+
 BuildRoot:        %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
 
 %if ( 0%{?rhel} && 0%{?rhel} <= 5 )
@@ -595,6 +597,7 @@
 
 %prep
 %setup -q -n %{name}-%{version}%{?prereltag}
+%patch0 -p1 -b .selinux
 
 %build
 # For whatever reason, install-sh is sometimes missing. When this gets fixed,
-----------------------------------------

$ cat SOURCES/glusterd-3.7.3-selinux.patch 
--- ./xlators/mgmt/glusterd/src/glusterd-quota.c_orig   2015-08-26 00:21:01.186000302 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-quota.c        2015-08-26 00:22:04.272000274 +0000
@@ -246,6 +246,7 @@
 
         runinit (&runner);
         runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                        "--selinux",
                          "-s", "localhost",
                          "--volfile-id", volname,
                         "--use-readdirp=no",
--- ./xlators/mgmt/glusterd/src/glusterd-volume-ops.c_orig      2015-08-26 00:21:01.195000301 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-volume-ops.c   2015-08-26 00:22:42.866000330 +0000
@@ -2770,7 +2770,7 @@
         runinit (&runner);
         glusterd_get_trusted_client_filepath (client_volfpath, volinfo,
                                       volinfo->transport_type);
-        runner_add_args (&runner, SBIN_DIR"/glusterfs", "-f", NULL);
+        runner_add_args (&runner, SBIN_DIR"/glusterfs", "-f", NULL, "--selinux");
         runner_argprintf (&runner, "%s", client_volfpath);
         runner_add_arg (&runner, "-l");
         runner_argprintf (&runner, DEFAULT_LOG_FILE_DIRECTORY
--- ./xlators/mgmt/glusterd/src/glusterd-mountbroker.c_orig     2015-08-26 00:21:01.185000302 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-mountbroker.c  2015-08-26 00:21:54.417000313 +0000
@@ -659,6 +659,7 @@
 
         runinit (&runner);
         runner_add_arg (&runner, SBIN_DIR"/glusterfs");
+        runner_add_arg (&runner, "--selinux");
         seq_dict_foreach (argdict, _runner_add, &runner);
         runner_add_arg (&runner, mtptemp);
         ret = runner_run_reuse (&runner);
--- ./xlators/mgmt/glusterd/src/glusterd-rebalance.c_orig       2015-08-26 00:21:01.188000301 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-rebalance.c    2015-08-26 00:22:14.668000295 +0000
@@ -261,6 +261,7 @@
 
         snprintf (volname, sizeof(volname), "rebalance/%s", volinfo->volname);
         runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                        "--selinux",
                          "-s", "localhost", "--volfile-id", volname,
                          "--xlator-option", "*dht.use-readdirp=yes",
                          "--xlator-option", "*dht.lookup-unhashed=yes",
--- ./xlators/mgmt/glusterd/src/glusterd-replace-brick.c_orig   2015-08-26 00:21:01.189000301 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-replace-brick.c        2015-08-26 00:22:18.535000291 +0000
@@ -83,6 +83,7 @@
 
         runinit (&runner);
         runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                        "--selinux",
                          "-s", "localhost",
                          "--volfile-id", volinfo->volname,
                          "--client-pid", pid,
--- ./xlators/mgmt/glusterd/src/glusterd-snapd-svc.c_orig       2015-08-26 00:21:01.191000299 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-snapd-svc.c    2015-08-26 00:22:22.317000303 +0000
@@ -287,6 +287,7 @@
 
         snprintf (snapd_id, sizeof (snapd_id), "snapd-%s", volinfo->volname);
         runner_add_args (&runner, SBIN_DIR"/glusterfsd",
+                        "--selinux",
                          "-s", svc->proc.volfileserver,
                          "--volfile-id", svc->proc.volfileid,
                          "-p", svc->proc.pidfile,
--- ./xlators/mgmt/glusterd/src/glusterd-utils.c_orig   2015-08-26 00:21:01.193000300 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-utils.c        2015-08-26 00:22:32.278000299 +0000
@@ -1620,6 +1620,7 @@
         (void) snprintf (glusterd_uuid, 1024, "*-posix.glusterd-uuid=%s",
                          uuid_utoa (MY_UUID));
         runner_add_args (&runner, SBIN_DIR"/glusterfsd",
+                        "--selinux",
                          "-s", brickinfo->hostname, "--volfile-id", volfile,
                          "-p", pidfile, "-S", socketpath,
                          "--brick-name", brickinfo->path,
--- ./xlators/mgmt/glusterd/src/glusterd-svc-mgmt.c_orig        2015-08-26 00:21:01.192000299 +0000
+++ ./xlators/mgmt/glusterd/src/glusterd-svc-mgmt.c     2015-08-26 00:22:28.821000303 +0000
@@ -182,6 +182,7 @@
         }
 
         runner_add_args (&runner, SBIN_DIR"/glusterfs",
+                        "--selinux",
                          "-s", svc->proc.volfileserver,
                          "--volfile-id", svc->proc.volfileid,
                          "-p", svc->proc.pidfile,
-----------------------------------------

Although this applies --selinux to all processes:

# ps -eo args |grep glust
/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid --selinux
/usr/sbin/glusterfsd --selinux -s ga --volfile-id gvol.ga.b1 -p /var/lib/glusterd/vols/gvol/run/ga-b1.pid -S /var/run/gluster/11753d16ee8a048e5f9b2331cbcfd4c7.socket --brick-name /b1 -l /var/log/glusterfs/bricks/b1.log --xlator-option *-posix.glusterd-uuid=6f491c3b-53d5-4928-8435-6c3d84f3ce53 --brick-port 49152 --xlator-option gvol-server.listen-port=49152
/usr/sbin/glusterfs --selinux -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/6502d8ef42d50130bd676cf9ef26c76d.socket --xlator-option *replicate*.node-uuid=6f491c3b-53d5-4928-8435-6c3d84f3ce53
/usr/sbin/glusterfs --selinux --volfile-server=localhost --volfile-id=/gvol /data

.. I still see the same error using chcon.  So something deeper in the code seems to be missing.

Anyone have an idea where the disconnect is?

Comment 2 Justin Clift 2015-11-26 15:54:18 UTC
Ping, this is being referenced externally as well:

https://github.com/CiscoCloud/microservices-infrastructure/issues/867#issuecomment-159730627

Comment 3 Niels de Vos 2016-06-30 10:51:45 UTC
Not that this bug depends on #1318100. Until that is resolved, Gluster will not be able to change the SELinux labels. The standard SELinux mount options to set specific labels for the whole mounted Gluster volume can be used (like "context", see #1287877 and http://review.gluster.org/12870).

https://bugzilla.redhat.com/show_bug.cgi?id=1318100#c1 also contains some more details, and points to a discussion on the mailinglist:

> At the moment it is not possible to set the SELinux context on a mounted
> Gluster Volume. We intend to have this functionality added to the Gluster core,
> and from there on add support to additional layers (FUSE, Labelled NFS, ...).
> 
> More details are listed in a conversation on the Gluster developers list:
>   http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/13071e

This is not something that we'll be able to backport to 3.7 or 3.8. We might be able to get initial support in GlusterFS 3.9. Changes to the kernel to support SELinux over FUSE might not be ready by that time though.

I'm closing this as DEFERRED, because SELinux support on Gluster volumes will not happen in 3.7.