Bug 852150 - mountbroker initiated umounts fail with EACCES on RHEL systems.
mountbroker initiated umounts fail with EACCES on RHEL systems.
Status: CLOSED DEFERRED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs-server (Show other bugs)
2.0
Unspecified Unspecified
medium Severity urgent
: ---
: ---
Assigned To: Venky Shankar
Sudhir D
:
Depends On: 811672
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-27 13:40 EDT by Vidya Sakar
Modified: 2013-03-03 21:07 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 811672
Environment:
Last Closed: 2013-02-26 07:14:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Vidya Sakar 2012-08-27 13:40:08 EDT
+++ This bug was initially created as a clone of Bug #811672 +++

Description of problem:

Mountbroker is a service provided by glusterd that can be used to request certain (pre-configured) glusterfs mounts and unmount them.

Mountbroker mounts succeed, but the unmount part fails with EACCES on RHEL-6.2 -- it seems to be caused by RHEL-specific security settings. Invoking umount(8) from shell (with same arguments as passed by glusterd) we succeed.

Version-Release number of selected component (if applicable):


How reproducible:

Deterministically.

Steps to Reproduce:

1. Set up mountbroker as described in RHS User Guide 9.2.5.2. (you may omit the creation of geogroup and the corresponding "option geo-replication-log-group geogroup" volume option)

2. mount the volume (here I use "slavevol", as in above doc) through mountbroker with following command:

# gluster system:: mount geoaccount user-map-root=geoaccount xlator-option=\*-dht.assert-no-child-down=true volfile-server=localhost volfile-id=slavevol client-pid=-1

This will give you back a path of the form /var/mountbroker-root/mb_hive/<mount id>

3. take down the above mount through mountbroker with following command:

# gluster system:: umount /var/mountbroker-root/mb_hive/<mount id>
  
Actual results:

We get back the message "umount failed" and the above command exits with 1.

Expected results:

We don't get any output and the above command exits with 0.

Additional info:

Stracing glusterd with following command:

# strace -s500 -f -eumount -p `cat /var/run/glusterd.pid`

displays

... umount("/var/mountbroker-root/mb_hive/mntTOKUsE", 0) = -1 EACCES (Permission denied)

--- Additional comment from csaba@redhat.com on 2012-04-11 12:59:48 EDT ---

For your ease, the RHS url:

http://docs.redhat.com/docs/en-US/Red_Hat_Storage/2/html/User_Guide/ch09s02s05s02.html

--- Additional comment from jdarcy@redhat.com on 2012-04-11 13:07:42 EDT ---

Anything in the audit log?

--- Additional comment from enakai@redhat.com on 2012-05-09 05:07:31 EDT ---

I suspect this is caused by SELinux. Here's the audit log.

-----
May  9 09:01:30 rhs20b2-02 kernel: type=1400 audit(1336554090.683:5): avc:  denied  { read } for  pid=2130 comm="umount" name="mnt48dS2M" dev=vda2 ino=29940 scontext=unconfined_u:system_r:mount_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=lnk_file
-----

And because of this, geo-replication with an unprivileged user, such as below, fails.

# gluster vol geo-replication vol01 geoaccount@rhs20b2-02::vol01_slave


A workaround is to "setenforce 0", but the final resolution should be an appropriate context labeling....

--- Additional comment from vbellur@redhat.com on 2012-05-14 03:31:11 EDT ---

Workaround available. Removing 3.3.0 milestone from this bug.

--- Additional comment from csaba@redhat.com on 2012-06-07 05:40:40 EDT ---

Etsuji: can you help us in dechipering the audit log entry? What is weird that the mount in question is created by glusterd (that is uid=0) and is to be taken down by glusterd too.

So it's a case of:
- root cannot do an umount
- the one who asked the system for some resource is not authorized to ask the system to release the resource

-- both seems contraintuitive from a naive POV. Which part of the context does imply this?

--- Additional comment from enakai@redhat.com on 2012-06-09 07:14:14 EDT ---

Ok... I did some investigation, and my hypothesis is:

- "umount" is forked from "glusterd", which is started by the upstart sequence.
- "mount" is forked from "gluterfs", which is forked from "glusterd".

Because of the difference of fork chain, "umount" and "mount" have differenct seculity contexts, and only "umount" is denied by SElinux.

Following is the result of my investigation which lead to the above hypothesis. 

====
1. After setting SELinux to Permissive mode and starting psacct on slave, I started georep from master.

2. According to the audit log on slave, "umount" has pid=1675, and its security context is "scontext=system_u:system_r:mount_t:s0".

Jun  9 10:37:56 slave01 kernel: type=1400 audit(1339238276.342:5): avc:  denied  { read } for  pid=1675 comm="umount" name="mnt3Tj3nM" dev=vda2 ino=30095 scontext=system_u:system_r:mount_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=lnk_file

3. According to the psacct log shown by "lastcomm --pid" on slave, the "umount" (pid=1675) is forked from pid=1217.

---------------
gluster               X georep01 __         0.00 secs Sat Jun  9 10:37     1671     1646
umount            S     root     __         0.00 secs Sat Jun  9 10:37     1675     1217
python                  georep01 __         0.03 secs Sat Jun  9 10:37     1673     1671
gluster               X georep01 __         0.00 secs Sat Jun  9 10:37     1656     1646
glusterfs         S     root     __         0.01 secs Sat Jun  9 10:37     1660     1217
mount             S     root     __         0.00 secs Sat Jun  9 10:37     1661     1660
python                  georep01 __         0.02 secs Sat Jun  9 10:37     1658     1656
---------------

And, pid=1271 is a "glusterd" launched by the upstart sequence, which has the security context "system_u:system_r:initrc_t:s0" as below.

# ps -efZ | grep 121[7]
system_u:system_r:initrc_t:s0   root      1217     1  0 10:30 ?        00:00:00 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid

4. On the otherhand, according to the same psacct log, "mount" (not denied) is forked from "glusterfs", not directly from "glusterd".
=====

To confirm the hypothesis, I stopped "glusterd" and started it again directly from the command line. Then "glusterd" has a different security context as below.

# ps -efZ | grep 1730
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 1730 1  1 10:45 ? 00:00:00 /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid

In this case, georep succeeded without any SElinux prevention.

--- Additional comment from vinaraya@redhat.com on 2012-06-24 21:12:37 EDT ---

This bug does not block 820843, removing the dependency.

Note You need to log in before you can comment on or make changes to this bug.