Bug 1256635 - Cannot set selinux context on files on a glusterfs mount
Cannot set selinux context on files on a glusterfs mount
Status: NEW
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: fuse (Show other bugs)
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
: Triaged, ZStream
: 811217 910380 1049066 1268296 (view as bug list)
Depends On: 1291606 1252627
Blocks: 1369781 817967
  Show dependency treegraph
Reported: 2015-08-25 03:19 EDT by Bipin Kunal
Modified: 2017-03-25 12:25 EDT (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
Red Hat Gluster Storage does not currently support SELinux Labeled mounts. On a FUSE mount, SELinux cannot currently distinguish file systems by subtype, and therefore cannot distinguish between different FUSE file systems (BZ#1291606). This means that a client-specific policy for Red Hat Gluster Storage cannot be defined, and SELinux cannot safely translate client-side extended attributes for files tracked by Red Hat Gluster Storage. A workaround is in progress for NFS-Ganesha mounts as part of BZ#1269584. When complete, BZ#1269584 will enable Red Hat Gluster Storage support for NFS version 4.2, including SELinux Labeled support.
Story Points: ---
Clone Of: 1252627
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
SPECS/kernel.spec.patch for kernel-2.6.32-504.el6.src.rpm (1.92 KB, patch)
2015-09-10 20:18 EDT, Bob Arendt
no flags Details | Diff
SOURCES/kernel-2.6.32-glusterselinux.patch for kernel-2.6.32-504.el6.src.rpm (5.17 KB, patch)
2015-09-10 20:20 EDT, Bob Arendt
no flags Details | Diff

  None (edit)
Description Bipin Kunal 2015-08-25 03:19:13 EDT
+++ This bug was initially created as a clone of Bug #1252627 +++

Description of problem:
After creating a gluster volume on top of xfs partitions, and mounting that volume, I'm unable to change the security context of files on the mounted filesystem.

Version-Release number of selected component:  Tested with both



How reproducible:  Always

Steps to Reproduce:  (for 3.7.3)
Built the latest RPM's from source from:

rpm -ivh glusterfs-3.7.3-1.el6.src.rpm
cd rpmbuild
rpmbuild -ba SPECS/glusterfs.spec |& tee log
  (put the rpm's in a private repository)

Create two test VM's, "ga" and "gb" using rhel66.  Each VM has partitions:
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda8        6558440 2802892   3415736  46% /
tmpfs             510028       0    510028   0% /dev/shm
/dev/vda3         200000   10400    189600   6% /b1
/dev/vda5         200000   10400    189600   6% /b2
/dev/vda6         200000   10400    189600   6% /b3
/dev/vda7         200000   10400    189600   6% /b4
/dev/vda1         243823   28113    202910  13% /boot

/dev/vda1: UUID="d3642293-57b1-4988-ac4f-85b0635e64c6" TYPE="ext4" 
/dev/vda2: UUID="f08a07bf-9222-45c7-9fd1-4d33207a8b86" TYPE="swap" 
/dev/vda3: UUID="ae1d0314-c22d-401e-a1b9-cf30fa4c6542" TYPE="xfs" 
/dev/vda5: UUID="5d32686c-5f00-4a00-ac06-539d2b85110d" TYPE="xfs" 
/dev/vda6: UUID="97e56854-76e6-4484-9afb-c5f56907df6e" TYPE="xfs" 
/dev/vda7: UUID="9e4d2bd4-de96-43f0-9aa3-3769cb22d508" TYPE="xfs" 
/dev/vda8: UUID="0be135b1-9bd5-49f5-8b21-8f43367a825b" TYPE="ext4" 

For the test we're only using the /b1 partitions on each host as our test bricks.

yum install glusterfs glusterfs-api glusterfs-cli glusterfs-client-xlators glusterfs-debuginfo glusterfs-fuse glusterfs-libs glusterfs-server

Create the a replicated volume:
chkconfig glusterd on
service glusterd start
gluster peer probe ga
gluster peer probe gb

gluster volume create gvol replica 2 transport tcp ga:/b1 gb:/b1 force
gluster volume start  gvol
gluster volume set    gvol auth.allow ga,gb
gluster volume set    gvol nfs.disable on

gluster volume info
    Volume Name: gvol
    Type: Replicate
    Volume ID: 4eeb493c-ed5f-4c3b-8945-4d14848a95d5
    Status: Started
    Number of Bricks: 1 x 2 = 2
    Transport-type: tcp
    Brick1: ga:/b1
    Brick2: gb:/b1
    Options Reconfigured:
    nfs.disable: on
    auth.allow: ga,gb
    performance.readdir-ahead: on

On each host, mount the volume
mkdir /data
mount -t glusterfs -o selinux localhost:/gvol /data

Check that the --selinux switch is asserted ... 
# ps -eo args |grep glust
/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid
/usr/sbin/glusterfsd -s gb --volfile-id gvol.gb.b1 -p /var/lib/glusterd/vols/gvol/run/gb-b1.pid -S /var/run/gluster/8dd23446126b2065164fdba21397998f.socket --brick-name /b1 -l /var/log/glusterfs/bricks/b1.log --xlator-option *-posix.glusterd-uuid=72fb7173-c1d3-4edd-b7e6-e7633e33358e --brick-port 49152 --xlator-option gvol-server.listen-port=49152
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a8c70c7c13620b79a8b5d26757294453.socket --xlator-option *replicate*.node-uuid=72fb7173-c1d3-4edd-b7e6-e7633e33358e
/usr/sbin/glusterfs --selinux --volfile-server=localhost --volfile-id=/gvol /data

Make some directories and files:

mkdir -p /data/a/b/c
echo test file > /data/a/b/myfile

Now for the test ...

Actual results:
[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:fusefs_t:s0    /data/a/b/myfile

[root@ga ~]# chcon -t tftpdir_rw_t /data/a/b/myfile
chcon: failed to change context of `/data/a/b/myfile' to `system_u:object_r:tftpdir_rw_t:s0': Operation not supported

[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:fusefs_t:s0    /data/a/b/myfile

Expected results:
[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:fusefs_t:s0    /data/a/b/myfile

[root@ga ~]# chcon -t tftpdir_rw_t /data/a/b/myfile

[root@ga ~]# ls -Z /data/a/b/myfile
-rw-r--r--. root root system_u:object_r:tftpdir_rw_t:s0    /data/a/b/myfile

Additional info:
Works on other file systems:
[root@ga ~]# touch /tmp/x
[root@ga ~]# ls -Z /tmp/x
-rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 /tmp/x
[root@ga ~]# chcon -t tftpdir_rw_t /tmp/x
[root@ga ~]# ls -Z /tmp/x
-rw-r--r--. root root unconfined_u:object_r:tftpdir_rw_t:s0 /tmp/x

Both hosts have selinux enabled in permissive mode

The mount has selinux capability enabled.  Is there anything on the server side that needs to be configured to enable selinux capability?
Comment 3 Bob Arendt 2015-08-26 20:21:05 EDT
I tried ensuring all processes had the --selinux flag (see bug 1252627#c1).  Recreating the volume from scratch with patched glusterfs-3.7.3 I saw the flags applied:

/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid --selinux
/usr/sbin/glusterfsd --selinux -s ga --volfile-id gvol.ga.b1 -p /var/lib/glusterd/vols/gvol/run/ga-b1.pid -S /var/run/gluster/11753d16ee8a048e5f9b2331cbcfd4c7.socket --brick-name /b1 -l /var/log/glusterfs/bricks/b1.log --xlator-option *-posix.glusterd-uuid=6f491c3b-53d5-4928-8435-6c3d84f3ce53 --brick-port 49152 --xlator-option gvol-server.listen-port=49152
/usr/sbin/glusterfs --selinux -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/6502d8ef42d50130bd676cf9ef26c76d.socket --xlator-option *replicate*.node-uuid=6f491c3b-53d5-4928-8435-6c3d84f3ce53
/usr/sbin/glusterfs --selinux --volfile-server=localhost --volfile-id=/gvol /data

But I received the same error:
# chcon -t tftpdir_rw_t /data/a/b/myfile
chcon: failed to change context of `/data/a/b/myfile' to `system_u:object_r:tftpdir_rw_t:s0': Operation not supported

Something deeper in the code seems to be missing. Does anyone have an idea where the disconnect is?
Comment 9 Bob Arendt 2015-09-04 12:06:35 EDT
This bug has NEEDINFO set - is there any information that I can provide?  The actual information need is not stated.

Also, has this been isolated to a particular portion of glusterfs?  If so, please share - I'd like to dive in and see if I get something working to meet my immediate needs.

Thank you.
Comment 10 Bipin Kunal 2015-09-07 01:57:49 EDT
Hi Bob,

Thanks for looking into the BZ. We have not yet isolated to a particular portion of glusterfs.

As of now we have figured out that this is somewhat due to one of the patch which was reverted long back:




We are thinking of adding this patch again into kernel and then test the behaviour.

We certainly need to dig deeper into the issue and need to fix it asap. Any help from you is really appreciated.

Bipin Kunal
Comment 11 Bipin Kunal 2015-09-10 08:22:03 EDT
Hey Niels,

I tried patching kernel 2.6.32-573.3.1 with the patch 
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=102aefdda4d8275ce7d7100bc16c88c74272b260  with some minor modification in the function prototype of "security_fs_use"

But still we are hitting same issue. Probably we need to look thoroughly into the code and find the gap.

Let me know if you your findings.

Bipin Kunal
Comment 12 Bob Arendt 2015-09-10 20:16:10 EDT
The previous patch doesn't apply to RHEL 6.  The API on security_fs_use() in security/selinux/ss/services.c is different.  The critical piece of the patch fails.

On RHEL6: (in security/selinux/ss/services.c)
  int security_fs_use(
      const char *fstype,
      unsigned int *behavior,
      u32 *sid)

  int security_fs_use(struct super_block *sb)

Looking at the patch from Anand Avati, almost ALL of it just adds diagnostic info to printk() statements.  The only piece that affects operation is in security/selinux/ss/services.c.  At this point he's searching for a policy match to sb->s_type (which is fstype) and sb->s_subtype.  But neither the subtype or superblock is available in the RHEL6 version of security_fs_use().

I took a different tack. It turns out that security_fs_use() is *only* called from selinux_get_mnt_opts() in security/selinux/hooks.c (the other file Anand modified).  It looks like an equivalent way to accomplish this was to create a concatenation of fstype and subtype and pass that in the fstype to security_fs_use().  The critical piece of the patch looks like this:

  char type_subtype[256];

  if (strcmp(sb->s_type->name, "proc") == 0) {
    sbsec->flags |= SE_SBPROC;
    strcpy(type_subtype, "proc");
  } else {
    int tlen;
    tlen = strlcpy(type_subtype, sb->s_type->name, sizeof(type_subtype));
    if (sb->s_subtype && sb->s_subtype[0] != '\0' && (tlen+2 < sizeof(type_subtype))) {
      type_subtype[tlen++] = '.';
      strlcpy(type_subtype+tlen, sb->s_subtype, sizeof(type_subtype)-tlen);
  rc = security_fs_use(type_subtype, &sbsec->behavior, &sbsec->sid);

I built this against kernel-2.6.32-504.el6.src.rpm, patching the SPECS/kernel.spec (kernel.spec.patch), which adds the patch SOURCES/kernel-2.6.32-glusterselinux.patch (both attached).

The build and kernel succeed, but we still see the original error.

Let's use systemtap for further diagnosis.  I made a simple systemtap script:

# cat g.stp
probe begin
  printf("Starting .. \n")

probe kernel.function("selinux_set_mnt_opts")
  printf("opts:sb.type = %s\n", kernel_string($sb->s_type->name))
  printf("opts:sb.HAS_SUBTYPE = %d\n", ($sb->s_type->fs_flags & 4))

  st = $sb->s_subtype
  if (st == 0) printf("opts:sb.s_subtype = NULL\n")
  else         printf("opts:sb.s_subtype = %s\n", kernel_string(st))

  op = $sb->s_options
  if (op == 0) printf("opts:sb.s_options = NULL\n")
  else         printf("opts:sb.s_options = %s\n", kernel_string(op))

probe kernel.function("security_fs_use")
  printf("  fs.beg:fstype   = %s\n", kernel_string($fstype))

probe kernel.function("security_fs_use").return
  printf("  fs.end:behavior = %s\n", kernel_string($behavior))
  printf("  fs.end:sid      = %x\n", $sid)

When I run it, and then mount my gluster volume:
  mount -t glusterfs -o selinux localhost:/gvol /data

[root@ga ~]# stap g.stp 
Starting .. 
opts:sb.type = fuse
opts:sb.HAS_SUBTYPE = 4
opts:sb.s_subtype = NULL
opts:sb.s_options = NULL
  fs.beg:fstype   = fuse
  fs.end:behavior = 
  fs.end:sid      = ffff88003b7113d8

Notice that even though the fs_flags bit for HAS_SUBTYPE is set, that the subtype field "opts:sb.subtype = NULL". The issue is that the subtype "glusterfs" is not added to sb->s_subtype record in the superblock.  But it is seen in /proc/mount
localhost:/gvol /data fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0

Any suggestions?  Maybe the superblock isn't filled out yet?  When does the subtype get set, and does glusterfs do this?
Comment 13 Bob Arendt 2015-09-10 20:18:48 EDT
Created attachment 1072402 [details]
SPECS/kernel.spec.patch for kernel-2.6.32-504.el6.src.rpm

Adds SOURCES/kernel-2.6.32-glusterselinux.patch
Comment 14 Bob Arendt 2015-09-10 20:20:32 EDT
Created attachment 1072403 [details]
SOURCES/kernel-2.6.32-glusterselinux.patch for kernel-2.6.32-504.el6.src.rpm

Use with SPECS/kernel.spec.patch
Comment 18 Bipin Kunal 2015-09-24 09:47:41 EDT
*** Bug 811217 has been marked as a duplicate of this bug. ***
Comment 19 Bipin Kunal 2015-09-24 09:53:54 EDT
*** Bug 1049066 has been marked as a duplicate of this bug. ***
Comment 20 Bipin Kunal 2015-09-24 09:55:10 EDT
*** Bug 910380 has been marked as a duplicate of this bug. ***
Comment 25 Iuliia Ievstignieieva 2016-01-15 03:22:27 EST
Hi all,

Could we please get some news on this bug?

Comment 28 Anjana Suparna Sriram 2016-01-21 05:57:43 EST
*** Bug 1268296 has been marked as a duplicate of this bug. ***
Comment 34 Csaba Henk 2017-02-22 07:39:09 EST
The feature depends on two kernel land work items.

There was work done towards both, but these efforts were not completed.

One is: a basic framework for SELinux attribute management
is needed in the FUSE VFS component to enable
FUSE servers to implement actual SELinux support.
Brian Foster's FUSE patches addressed this:


Other is: fs subtype support for SELinux in order
to allow SELinux policies to be fine-tuned to
distinguish between various FUSE file systems
(ie. be able to reward SELinux supporting FUSE
fs-es with full feature set). Avati had a patch
for this that was even merged:


but it proved to be problematic and thus
reverted (in two separate branches):


Until these items are picked up and taken to completion,
*and* they are backported to downstream kernel, work on
GlusterFS side won't have much effect, so this
issue cannot be realistically scheduled.

Note You need to log in before you can comment on or make changes to this bug.