Bug 811672

Summary: mountbroker initiated umounts fail with EACCES on RHEL systems.
Product: [Community] GlusterFS Reporter: Csaba Henk <csaba>
Component: glusterdAssignee: Venky Shankar <vshankar>
Status: CLOSED DEFERRED QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: mainlineCC: enakai, gluster-bugs, jdarcy, vbellur, vinaraya, vkoppad
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 820843 852150 (view as bug list) Environment:
Last Closed: 2013-02-26 07:13:30 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 852150    

Description Csaba Henk 2012-04-11 12:57:29 EDT
Description of problem:

Mountbroker is a service provided by glusterd that can be used to request certain (pre-configured) glusterfs mounts and unmount them.

Mountbroker mounts succeed, but the unmount part fails with EACCES on RHEL-6.2 -- it seems to be caused by RHEL-specific security settings. Invoking umount(8) from shell (with same arguments as passed by glusterd) we succeed.

Version-Release number of selected component (if applicable):


How reproducible:

Deterministically.

Steps to Reproduce:

1. Set up mountbroker as described in RHS User Guide 9.2.5.2. (you may omit the creation of geogroup and the corresponding "option geo-replication-log-group geogroup" volume option)

2. mount the volume (here I use "slavevol", as in above doc) through mountbroker with following command:

# gluster system:: mount geoaccount user-map-root=geoaccount xlator-option=\*-dht.assert-no-child-down=true volfile-server=localhost volfile-id=slavevol client-pid=-1

This will give you back a path of the form /var/mountbroker-root/mb_hive/<mount id>

3. take down the above mount through mountbroker with following command:

# gluster system:: umount /var/mountbroker-root/mb_hive/<mount id>
  
Actual results:

We get back the message "umount failed" and the above command exits with 1.

Expected results:

We don't get any output and the above command exits with 0.

Additional info:

Stracing glusterd with following command:

# strace -s500 -f -eumount -p `cat /var/run/glusterd.pid`

displays

... umount("/var/mountbroker-root/mb_hive/mntTOKUsE", 0) = -1 EACCES (Permission denied)
Comment 1 Csaba Henk 2012-04-11 12:59:48 EDT
For your ease, the RHS url:

http://docs.redhat.com/docs/en-US/Red_Hat_Storage/2/html/User_Guide/ch09s02s05s02.html
Comment 2 Jeff Darcy 2012-04-11 13:07:42 EDT
Anything in the audit log?
Comment 3 Etsuji Nakai 2012-05-09 05:07:31 EDT
I suspect this is caused by SELinux. Here's the audit log.

-----
May  9 09:01:30 rhs20b2-02 kernel: type=1400 audit(1336554090.683:5): avc:  denied  { read } for  pid=2130 comm="umount" name="mnt48dS2M" dev=vda2 ino=29940 scontext=unconfined_u:system_r:mount_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=lnk_file
-----

And because of this, geo-replication with an unprivileged user, such as below, fails.

# gluster vol geo-replication vol01 geoaccount@rhs20b2-02::vol01_slave


A workaround is to "setenforce 0", but the final resolution should be an appropriate context labeling....
Comment 4 Vijay Bellur 2012-05-14 03:31:11 EDT
Workaround available. Removing 3.3.0 milestone from this bug.
Comment 5 Csaba Henk 2012-06-07 05:40:40 EDT
Etsuji: can you help us in dechipering the audit log entry? What is weird that the mount in question is created by glusterd (that is uid=0) and is to be taken down by glusterd too.

So it's a case of:
- root cannot do an umount
- the one who asked the system for some resource is not authorized to ask the system to release the resource

-- both seems contraintuitive from a naive POV. Which part of the context does imply this?
Comment 6 Etsuji Nakai 2012-06-09 07:14:14 EDT
Ok... I did some investigation, and my hypothesis is:

- "umount" is forked from "glusterd", which is started by the upstart sequence.
- "mount" is forked from "gluterfs", which is forked from "glusterd".

Because of the difference of fork chain, "umount" and "mount" have differenct seculity contexts, and only "umount" is denied by SElinux.

Following is the result of my investigation which lead to the above hypothesis. 

====
1. After setting SELinux to Permissive mode and starting psacct on slave, I started georep from master.

2. According to the audit log on slave, "umount" has pid=1675, and its security context is "scontext=system_u:system_r:mount_t:s0".

Jun  9 10:37:56 slave01 kernel: type=1400 audit(1339238276.342:5): avc:  denied  { read } for  pid=1675 comm="umount" name="mnt3Tj3nM" dev=vda2 ino=30095 scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=lnk_file

3. According to the psacct log shown by "lastcomm --pid" on slave, the "umount" (pid=1675) is forked from pid=1217.

---------------
gluster               X georep01 __         0.00 secs Sat Jun  9 10:37     1671     1646
umount            S     root     __         0.00 secs Sat Jun  9 10:37     1675     1217
python                  georep01 __         0.03 secs Sat Jun  9 10:37     1673     1671
gluster               X georep01 __         0.00 secs Sat Jun  9 10:37     1656     1646
glusterfs         S     root     __         0.01 secs Sat Jun  9 10:37     1660     1217
mount             S     root     __         0.00 secs Sat Jun  9 10:37     1661     1660
python                  georep01 __         0.02 secs Sat Jun  9 10:37     1658     1656
---------------

And, pid=1271 is a "glusterd" launched by the upstart sequence, which has the security context "system_u:system_r:initrc_t:s0" as below.

# ps -efZ | grep 121[7]
system_u:system_r:initrc_t:s0   root      1217     1  0 10:30 ?        00:00:00 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid

4. On the otherhand, according to the same psacct log, "mount" (not denied) is forked from "glusterfs", not directly from "glusterd".
=====

To confirm the hypothesis, I stopped "glusterd" and started it again directly from the command line. Then "glusterd" has a different security context as below.

# ps -efZ | grep 1730
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 1730 1  1 10:45 ? 00:00:00 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid

In this case, georep succeeded without any SElinux prevention.
Comment 7 Vidya Sakar 2012-06-24 21:12:37 EDT
This bug does not block 820843, removing the dependency.