+++ This bug was initially created as a clone of Bug #1252627 +++ Description of problem: After creating a gluster volume on top of xfs partitions, and mounting that volume, I'm unable to change the security context of files on the mounted filesystem. Version-Release number of selected component: Tested with both glusterfs-3.6.4-1.el6.x86_64 glusterfs-api-3.6.4-1.el6.x86_64 glusterfs-cli-3.6.4-1.el6.x86_64 glusterfs-debuginfo-3.6.4-1.el6.x86_64 glusterfs-extra-xlators-3.6.4-1.el6.x86_64 glusterfs-fuse-3.6.4-1.el6.x86_64 glusterfs-libs-3.6.4-1.el6.x86_64 glusterfs-server-3.6.4-1.el6.x86_64 glusterfs-3.7.3-1.el6.x86_64 glusterfs-api-3.7.3-1.el6.x86_64 glusterfs-cli-3.7.3-1.el6.x86_64 glusterfs-client-xlators-3.7.3-1.el6.x86_64 glusterfs-debuginfo-3.7.3-1.el6.x86_64 glusterfs-fuse-3.7.3-1.el6.x86_64 glusterfs-libs-3.7.3-1.el6.x86_64 glusterfs-server-3.7.3-1.el6.x86_64 How reproducible: Always Steps to Reproduce: (for 3.7.3) Built the latest RPM's from source from: http://dl.fedoraproject.org/pub/epel/6/SRPMS/userspace-rcu-0.7.7-1.el6.src.rpm http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/epel-6.6/SRPMS/glusterfs-3.7.3-1.el6.src.rpm rpm -ivh glusterfs-3.7.3-1.el6.src.rpm cd rpmbuild rpmbuild -ba SPECS/glusterfs.spec |& tee log (put the rpm's in a private repository) Create two test VM's, "ga" and "gb" using rhel66. Each VM has partitions: Filesystem 1K-blocks Used Available Use% Mounted on /dev/vda8 6558440 2802892 3415736 46% / tmpfs 510028 0 510028 0% /dev/shm /dev/vda3 200000 10400 189600 6% /b1 /dev/vda5 200000 10400 189600 6% /b2 /dev/vda6 200000 10400 189600 6% /b3 /dev/vda7 200000 10400 189600 6% /b4 /dev/vda1 243823 28113 202910 13% /boot /dev/vda1: UUID="d3642293-57b1-4988-ac4f-85b0635e64c6" TYPE="ext4" /dev/vda2: UUID="f08a07bf-9222-45c7-9fd1-4d33207a8b86" TYPE="swap" /dev/vda3: UUID="ae1d0314-c22d-401e-a1b9-cf30fa4c6542" TYPE="xfs" /dev/vda5: UUID="5d32686c-5f00-4a00-ac06-539d2b85110d" TYPE="xfs" /dev/vda6: UUID="97e56854-76e6-4484-9afb-c5f56907df6e" TYPE="xfs" /dev/vda7: UUID="9e4d2bd4-de96-43f0-9aa3-3769cb22d508" TYPE="xfs" /dev/vda8: UUID="0be135b1-9bd5-49f5-8b21-8f43367a825b" TYPE="ext4" For the test we're only using the /b1 partitions on each host as our test bricks. yum install glusterfs glusterfs-api glusterfs-cli glusterfs-client-xlators glusterfs-debuginfo glusterfs-fuse glusterfs-libs glusterfs-server Create the a replicated volume: chkconfig glusterd on service glusterd start gluster peer probe ga gluster peer probe gb gluster volume create gvol replica 2 transport tcp ga:/b1 gb:/b1 force gluster volume start gvol gluster volume set gvol auth.allow ga,gb gluster volume set gvol nfs.disable on gluster volume info Volume Name: gvol Type: Replicate Volume ID: 4eeb493c-ed5f-4c3b-8945-4d14848a95d5 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: ga:/b1 Brick2: gb:/b1 Options Reconfigured: nfs.disable: on auth.allow: ga,gb performance.readdir-ahead: on On each host, mount the volume mkdir /data mount -t glusterfs -o selinux localhost:/gvol /data Check that the --selinux switch is asserted ... # ps -eo args |grep glust /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid /usr/sbin/glusterfsd -s gb --volfile-id gvol.gb.b1 -p /var/lib/glusterd/vols/gvol/run/gb-b1.pid -S /var/run/gluster/8dd23446126b2065164fdba21397998f.socket --brick-name /b1 -l /var/log/glusterfs/bricks/b1.log --xlator-option *-posix.glusterd-uuid=72fb7173-c1d3-4edd-b7e6-e7633e33358e --brick-port 49152 --xlator-option gvol-server.listen-port=49152 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/a8c70c7c13620b79a8b5d26757294453.socket --xlator-option *replicate*.node-uuid=72fb7173-c1d3-4edd-b7e6-e7633e33358e /usr/sbin/glusterfs --selinux --volfile-server=localhost --volfile-id=/gvol /data Make some directories and files: mkdir -p /data/a/b/c echo test file > /data/a/b/myfile Now for the test ... Actual results: [root@ga ~]# ls -Z /data/a/b/myfile -rw-r--r--. root root system_u:object_r:fusefs_t:s0 /data/a/b/myfile [root@ga ~]# chcon -t tftpdir_rw_t /data/a/b/myfile chcon: failed to change context of `/data/a/b/myfile' to `system_u:object_r:tftpdir_rw_t:s0': Operation not supported [root@ga ~]# ls -Z /data/a/b/myfile -rw-r--r--. root root system_u:object_r:fusefs_t:s0 /data/a/b/myfile Expected results: [root@ga ~]# ls -Z /data/a/b/myfile -rw-r--r--. root root system_u:object_r:fusefs_t:s0 /data/a/b/myfile [root@ga ~]# chcon -t tftpdir_rw_t /data/a/b/myfile [root@ga ~]# ls -Z /data/a/b/myfile -rw-r--r--. root root system_u:object_r:tftpdir_rw_t:s0 /data/a/b/myfile Additional info: Works on other file systems: [root@ga ~]# touch /tmp/x [root@ga ~]# ls -Z /tmp/x -rw-r--r--. root root unconfined_u:object_r:user_tmp_t:s0 /tmp/x [root@ga ~]# chcon -t tftpdir_rw_t /tmp/x [root@ga ~]# ls -Z /tmp/x -rw-r--r--. root root unconfined_u:object_r:tftpdir_rw_t:s0 /tmp/x Both hosts have selinux enabled in permissive mode The mount has selinux capability enabled. Is there anything on the server side that needs to be configured to enable selinux capability?
I tried ensuring all processes had the --selinux flag (see bug 1252627#c1). Recreating the volume from scratch with patched glusterfs-3.7.3 I saw the flags applied: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid --selinux /usr/sbin/glusterfsd --selinux -s ga --volfile-id gvol.ga.b1 -p /var/lib/glusterd/vols/gvol/run/ga-b1.pid -S /var/run/gluster/11753d16ee8a048e5f9b2331cbcfd4c7.socket --brick-name /b1 -l /var/log/glusterfs/bricks/b1.log --xlator-option *-posix.glusterd-uuid=6f491c3b-53d5-4928-8435-6c3d84f3ce53 --brick-port 49152 --xlator-option gvol-server.listen-port=49152 /usr/sbin/glusterfs --selinux -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/6502d8ef42d50130bd676cf9ef26c76d.socket --xlator-option *replicate*.node-uuid=6f491c3b-53d5-4928-8435-6c3d84f3ce53 /usr/sbin/glusterfs --selinux --volfile-server=localhost --volfile-id=/gvol /data But I received the same error: # chcon -t tftpdir_rw_t /data/a/b/myfile chcon: failed to change context of `/data/a/b/myfile' to `system_u:object_r:tftpdir_rw_t:s0': Operation not supported Something deeper in the code seems to be missing. Does anyone have an idea where the disconnect is?
This bug has NEEDINFO set - is there any information that I can provide? The actual information need is not stated. Also, has this been isolated to a particular portion of glusterfs? If so, please share - I'd like to dive in and see if I get something working to meet my immediate needs. Thank you.
Hi Bob, Thanks for looking into the BZ. We have not yet isolated to a particular portion of glusterfs. As of now we have figured out that this is somewhat due to one of the patch which was reverted long back: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4d546f81717d253ab67643bf072c6d8821a9249c or http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=29b1deb2a48a9dd02b93597aa4c055a24c0e989f We are thinking of adding this patch again into kernel and then test the behaviour. We certainly need to dig deeper into the issue and need to fix it asap. Any help from you is really appreciated. Thanks, Bipin Kunal
Hey Niels, I tried patching kernel 2.6.32-573.3.1 with the patch http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=102aefdda4d8275ce7d7100bc16c88c74272b260 with some minor modification in the function prototype of "security_fs_use" But still we are hitting same issue. Probably we need to look thoroughly into the code and find the gap. Let me know if you your findings. Bipin Kunal
The previous patch doesn't apply to RHEL 6. The API on security_fs_use() in security/selinux/ss/services.c is different. The critical piece of the patch fails. On RHEL6: (in security/selinux/ss/services.c) int security_fs_use( const char *fstype, unsigned int *behavior, u32 *sid) On RHEL7: int security_fs_use(struct super_block *sb) Looking at the patch from Anand Avati, almost ALL of it just adds diagnostic info to printk() statements. The only piece that affects operation is in security/selinux/ss/services.c. At this point he's searching for a policy match to sb->s_type (which is fstype) and sb->s_subtype. But neither the subtype or superblock is available in the RHEL6 version of security_fs_use(). I took a different tack. It turns out that security_fs_use() is *only* called from selinux_get_mnt_opts() in security/selinux/hooks.c (the other file Anand modified). It looks like an equivalent way to accomplish this was to create a concatenation of fstype and subtype and pass that in the fstype to security_fs_use(). The critical piece of the patch looks like this: char type_subtype[256]; if (strcmp(sb->s_type->name, "proc") == 0) { sbsec->flags |= SE_SBPROC; strcpy(type_subtype, "proc"); } else { int tlen; tlen = strlcpy(type_subtype, sb->s_type->name, sizeof(type_subtype)); if (sb->s_subtype && sb->s_subtype[0] != '\0' && (tlen+2 < sizeof(type_subtype))) { type_subtype[tlen++] = '.'; strlcpy(type_subtype+tlen, sb->s_subtype, sizeof(type_subtype)-tlen); } } rc = security_fs_use(type_subtype, &sbsec->behavior, &sbsec->sid); I built this against kernel-2.6.32-504.el6.src.rpm, patching the SPECS/kernel.spec (kernel.spec.patch), which adds the patch SOURCES/kernel-2.6.32-glusterselinux.patch (both attached). The build and kernel succeed, but we still see the original error. Let's use systemtap for further diagnosis. I made a simple systemtap script: ---------------- # cat g.stp probe begin { printf("Starting .. \n") } probe kernel.function("selinux_set_mnt_opts") { printf("opts:sb.type = %s\n", kernel_string($sb->s_type->name)) printf("opts:sb.HAS_SUBTYPE = %d\n", ($sb->s_type->fs_flags & 4)) st = $sb->s_subtype if (st == 0) printf("opts:sb.s_subtype = NULL\n") else printf("opts:sb.s_subtype = %s\n", kernel_string(st)) op = $sb->s_options if (op == 0) printf("opts:sb.s_options = NULL\n") else printf("opts:sb.s_options = %s\n", kernel_string(op)) } probe kernel.function("security_fs_use") { printf(" fs.beg:fstype = %s\n", kernel_string($fstype)) } probe kernel.function("security_fs_use").return { printf(" fs.end:behavior = %s\n", kernel_string($behavior)) printf(" fs.end:sid = %x\n", $sid) } ---------------- When I run it, and then mount my gluster volume: mount -t glusterfs -o selinux localhost:/gvol /data [root@ga ~]# stap g.stp Starting .. opts:sb.type = fuse opts:sb.HAS_SUBTYPE = 4 opts:sb.s_subtype = NULL opts:sb.s_options = NULL fs.beg:fstype = fuse fs.end:behavior = fs.end:sid = ffff88003b7113d8 Notice that even though the fs_flags bit for HAS_SUBTYPE is set, that the subtype field "opts:sb.subtype = NULL". The issue is that the subtype "glusterfs" is not added to sb->s_subtype record in the superblock. But it is seen in /proc/mount localhost:/gvol /data fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 0 0 Any suggestions? Maybe the superblock isn't filled out yet? When does the subtype get set, and does glusterfs do this?
Created attachment 1072402 [details] SPECS/kernel.spec.patch for kernel-2.6.32-504.el6.src.rpm Adds SOURCES/kernel-2.6.32-glusterselinux.patch
Created attachment 1072403 [details] SOURCES/kernel-2.6.32-glusterselinux.patch for kernel-2.6.32-504.el6.src.rpm Use with SPECS/kernel.spec.patch
*** Bug 811217 has been marked as a duplicate of this bug. ***
*** Bug 1049066 has been marked as a duplicate of this bug. ***
*** Bug 910380 has been marked as a duplicate of this bug. ***
Hi all, Could we please get some news on this bug? Thanks
*** Bug 1268296 has been marked as a duplicate of this bug. ***
The feature depends on two kernel land work items. There was work done towards both, but these efforts were not completed. One is: a basic framework for SELinux attribute management is needed in the FUSE VFS component to enable FUSE servers to implement actual SELinux support. Brian Foster's FUSE patches addressed this: https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/1385389343-55663-1-git-send-email-bfoster%40redhat.com/#msg31678712 or http://fuse.996288.n3.nabble.com/PATCH-v2-0-3-fuse-support-selinux-extended-attributes-tt11871.html Other is: fs subtype support for SELinux in order to allow SELinux policies to be fine-tuned to distinguish between various FUSE file systems (ie. be able to reward SELinux supporting FUSE fs-es with full feature set). Avati had a patch for this that was even merged: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=102aefdda4d8275ce7d7100bc16c88c74272b260 but it proved to be problematic and thus reverted (in two separate branches): http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4d546f81717d253ab67643bf072c6d8821a9249c http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=29b1deb2a48a9dd02b93597aa4c055a24c0e989f Until these items are picked up and taken to completion, *and* they are backported to downstream kernel, work on GlusterFS side won't have much effect, so this issue cannot be realistically scheduled.
Based on the technical summary given in Comment 34: This is an elongated issue with various "WiP" bits floating around, but we are not able to provide a fix at the moment and the foreseeable future. More precisely, it's technically possible to fix it, but that involves a significant amount of kernel work, which is not really feasible to decide and schedule within RGHS procuct / Storage BU staff scope. I'm asking PM for help to find out where to take this forward.
In the last 1 year we didn't get any further on this, and as stated, this needs lot of work in kernel side to get this going! Would take this as a CLOSED-UPSTREAM to highlight we would fix it in upstream someday! Would like to get opinion on this!
Hello All, has there been any progress in this issue? Suzu