Bug 1321786

Summary: [SELinux] volume not getting exported after setting the option ganesha.enable -RHEL7
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Shashank Raj <sraj>
Component: nfs-ganeshaAssignee: Bug Updates Notification Mailing List <rhs-bugs>
Status: CLOSED ERRATA QA Contact: Shashank Raj <sraj>
Severity: urgent Docs Contact: Marie Hornickova <mdolezel>
Priority: unspecified    
Version: rhgs-3.1CC: akhakhar, asrivast, bugs, jthottan, kkeithle, mdolezel, ndevos, nlevinki, pprakash, rhinduja, sashinde, skoduri, sraj
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
When the nfs-ganesha server was set up on four nodes with 2X2 volume, the volume was not exported after the nfs-ganesha service was enabled. With this update, a workaround has been provided which ensures that the volume is exported as expected in the described scenario.
 Story Points: ---
Clone Of: 1311911 Environment:
Last Closed: 2016-06-23 05:34:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1312809    
Bug Blocks: 1311817    

Description Shashank Raj 2016-03-29 07:23:10 UTC 
+++ This bug was initially created as a clone of Bug #1311911 +++

Description of problem:
volume not getting exported after setting the option ganesha.enable 

Version-Release number of selected component (if applicable):
glusterfs-ganesha-3.7.8-1.el7.x86_64
nfs-ganesha-2.2.0-12.el6rhs.x86_64
glusterfs-3.7.8-1.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Setup nfs-ganesha on 4 nodes
2. Create a 2X2 volume.
3. Start the volume
4. set the volume option- ganesha.enable on. it says success, but the volume is actually not exported

Export file is present
[root@dhcp46-59 ~]# cat /etc/ganesha/exports/export.testvol.conf 
# WARNING : Using Gluster CLI will overwrite manual
# changes made to this file. To avoid it, edit the
# file and run ganesha-ha.sh --refresh-config.
EXPORT{
      Export_Id= 2 ;
      Path = "/testvol";
      FSAL {
           name = GLUSTER;
           hostname="localhost";
          volume="testvol";
           }
      Access_type = RW;
      Disable_ACL = true;
      Squash="No_root_squash";
      Pseudo="/testvol";
      Protocols = "3", "4" ;
      Transports = "UDP","TCP";
      SecType = "sys";
     }


Also ganesha.conf file has entry of this config file:
[root@dhcp46-59 ~]# cat /etc/ganesha/ganesha.conf 
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#

#EXPORT
#{
	# Export Id (mandatory, each EXPORT must have a unique Export_Id)
#	Export_Id = 77;

	# Exported path (mandatory)
#	Path = "/testvol";

	# Pseudo Path (required for NFS v4)
#	Pseudo = "/testvol";

	# Required for access (default is None)
	# Could use CLIENT blocks instead
#	Access_Type = RW;

	# Allow root access
#	Squash = No_Root_Squash;

	# Security flavor supported
#	SecType = "sys";

	# Exporting FSAL
#	FSAL {
#		Name = "GLUSTER";
#		Hostname = localhost;
#		Volume = "testvol";
#	}
#}
#######################################################
#Create this export block in a file which has the following parameters
#in the global part. Or create a separate file with the export block
#and include in the following block.

NFS_Core_Param {
        #Use supplied name other tha IP In NSM operations
        NSM_Use_Caller_Name = true;
        #Copy lock states into "/var/lib/nfs/ganesha" dir
        Clustered = false;
        #Use a non-privileged port for RQuota
        Rquota_Port = 4501;
        MNT_Port = 20048;
        NLM_Port = 32000;
}

%include "/etc/ganesha/exports/export.vol.conf


But showmount does not show that volume is exported

Actual results:showmount does not show that volume is exported


Expected results: on setting ganesha.enable option volume should get exported


Additional info:

--- Additional comment from Apeksha on 2016-02-25 05:23:18 EST ---

After Restarting the nfs-ganesha service on all the nodes, the volume is getting exported

--- Additional comment from Jiffin on 2016-02-26 14:10:14 EST ---

IMO the issue may be related to selinux policies, in the audit log the following logs can found while enable and disabling the ganesha.enable option

type=USER_AVC msg=audit(1456522097.022:4933): pid=902 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.ganesha.nfsd.exportmgr member=AddExport dest=org.ganesha.nfsd spid=10631 tpid=26644 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'

type=USER_AVC msg=audit(1456521684.235:4932): pid=902 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.ganesha.nfsd.exportmgr member=RemoveExport dest=org.ganesha.nfsd spid=5403 tpid=26644 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?

I ran the wrapper script (/usr/libexec/dbus-send.sh) used by cli from the terminal with necessary parameters, the volume got exported.
for example
/usr/libexec/ganesha/dbus-send.sh /etc/ganesha/ <on/off> <volume name>

--- Additional comment from Apeksha on 2016-02-29 04:45:27 EST ---

**Steps when selinux was in enforcing mode


    [root@dhcp46-59 ~]# getenforce
    Enforcing
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]# gluster v create rs 10.70.46.59:/root/brick2 force
    volume create: rs: success: please start the volume to access data
    [root@dhcp46-59 ~]# gluster v start rs
    volume start: rs: success
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]# #gluster v set rs ganesha.enable on
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]# grep -i "avc" /var/log/audit/audit.log
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]# gluster v set rs ganesha.enable on
    volume set: success
    [root@dhcp46-59 ~]# grep -i "avc" /var/log/audit/audit.log
    type=USER_AVC msg=audit(1456767046.846:5613): pid=902 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=signal interface=org.ganesha.nfsd.exportmgr member=AddExport dest=org.ganesha.nfsd spid=1613 tpid=26644 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
    [root@dhcp46-59 ~]# cat /var/log/audit/audit.log | audit2allow
     
     
    #============= glusterd_t ==============
    allow glusterd_t initrc_t:dbus send_msg;
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]# showmount -e localhost
    Export list for localhost:
    /sample (everyone)
    [root@dhcp46-59 ~]#



**Steps when selinux is in permissive mode


    [root@dhcp46-59 ~]# setenforce 0
    [root@dhcp46-59 ~]# gluster v create rs1 10.70.46.59:/root/brick3 force
    volume create: rs1: success: please start the volume to access data
    [root@dhcp46-59 ~]# gluster v start rs1
    volume start: rs1: success
    [root@dhcp46-59 ~]# gluster v set rs1 ganesha.enable on
    volume set: success
    [root@dhcp46-59 ~]# showmount -e localhost
    Export list for localhost:
    /sample (everyone)
    /rs1    (everyone)
    [root@dhcp46-59 ~]# grep -i "avc" /var/log/audit/audit.log
    type=USER_AVC msg=audit(1456767046.846:5613): pid=902 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=signal interface=org.ganesha.nfsd.exportmgr member=AddExport dest=org.ganesha.nfsd spid=1613 tpid=26644 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
    type=USER_AVC msg=audit(1456767084.524:5622): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=0)  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
    type=USER_AVC msg=audit(1456767110.891:5623): pid=902 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=signal interface=org.ganesha.nfsd.exportmgr member=AddExport dest=org.ganesha.nfsd spid=2540 tpid=26644 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
    [root@dhcp46-59 ~]# cat /var/log/audit/audit.log | audit2allow
     
     
    #============= glusterd_t ==============
    allow glusterd_t initrc_t:dbus send_msg;
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]#
    [root@dhcp46-59 ~]# rpm -qa | grep selinux-policy
    selinux-policy-3.13.1-60.el7_2.3.noarch
    selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
    [root@dhcp46-59 ~]#

--- Additional comment from Shashank Raj on 2016-03-28 11:45:33 EDT ---

Observed the same issue with 3.1.3 build (3.7.9-1) as well where in volume doesn't get exported after setting ganesha.enable on 

[root@dhcp46-247 brick0]# gluster volume set testvol ganesha.enable on
volume set: success

and it shows below user avc in audit.log

type=USER_AVC msg=audit(1459163604.191:3776): pid=654 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_call interface=org.ganesha.nfsd.exportmgr member=RemoveExport dest=org.ganesha.nfsd spid=27599 tpid=28904 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'

[root@dhcp46-247 ~]# rpm -qa|grep selinux

selinux-policy-targeted-3.13.1-60.el7.noarch
selinux-policy-3.13.1-60.el7.noarch

--- Additional comment from Shashank Raj on 2016-03-28 12:04:21 EDT ---

In permissive mode:

[root@dhcp46-247 exports]# setenforce 0
[root@dhcp46-247 exports]# getenforce
Permissive

[root@dhcp46-247 exports]# gluster volume list
gluster_shared_storage
newvol

[root@dhcp46-247 exports]# gluster volume set newvol ganesha.enable on
volume set: success

[root@dhcp46-247 exports]# showmount -e localhost
Export list for localhost:
/newvol (everyone)

Below messages in audit.log

type=MAC_STATUS msg=audit(1459179476.783:3983): enforcing=0 old_enforcing=1 auid=0 ses=182
type=SYSCALL msg=audit(1459179476.783:3983): arch=c000003e syscall=1 success=yes exit=1 a0=3 a1=7ffe1ea8ef10 a2=1 a3=7ffe1ea8ec90 items=0 ppid=8970 pid=13164 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=182 comm="setenforce" exe="/usr/sbin/setenforce" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)

type=USER_AVC msg=audit(1459179552.967:3984): pid=654 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=signal interface=org.ganesha.nfsd.exportmgr member=AddExport dest=org.ganesha.nfsd spid=13573 tpid=28904 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
Comment 7 Shashank Raj 2016-05-11 06:58:37 UTC 
Verified this bug with the selinux-policy-3.13.1-60.el7_2.4.noarch and nfs-ganesha-2.3.1-6.el7rhgs.x86_64, and its working as expected.

After upgrading to the above selinux policy, volumes can be exported through ganesha and no denial AVC's are seen in audit.log

Based on the above observation, marking this bug as Verified.
Comment 9 errata-xmlrpc 2016-06-23 05:34:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2016:1247