Bug 1323947

Summary: [SELinux]: AVC's denying permissions related to statd, observed in nfs-ganesha environment -RHEL7
Product: Red Hat Enterprise Linux 7 Reporter: Shashank Raj <sraj>
Component: selinux-policyAssignee: Lukas Vrabec <lvrabec>
Status: CLOSED ERRATA QA Contact: Milos Malik <mmalik>
Severity: high Docs Contact: Marie Hornickova <mdolezel>
Priority: urgent    
Version: 7.2CC: james.christensen, jthottan, kkeithle, lvrabec, mdolezel, mgrepl, mkolaja, mmalik, ndevos, nlevinki, plautrba, pprakash, pvrabec, rhs-bugs, sashinde, skoduri, sraj, ssekidde, storage-qa-internal
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: selinux-policy-3.13.1-70.el7 Doc Type: Bug Fix
Doc Text:
Due to missing rules in the Gluster SELinux policy, the nfs-ganesha service failed to connect to the rpc.statd daemon after a node reboot in the situation where the nfs-ganesha server was installed on four nodes. The underlying code has been fixed, and nfs-ganesha no longer fails in the described scenario.
Story Points: ---
Clone Of: 1323740
: 1332577 1333875 (view as bug list) Environment:
Last Closed: 2016-11-04 02:46:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1323740, 1332577, 1333875    

Description Shashank Raj 2016-04-05 07:09:26 UTC
Created attachment 1143678 [details]
audit logs

+++ This bug was initially created as a clone of Bug #1323740 +++

Description of problem:
nfs-ganesha.service status shows "failed to connect to statd" after node reboot

Version-Release number of selected component (if applicable):
glusterfs-3.7.9-1

How reproducible:
twice

Steps to Reproduce:
1.Configure nfs-ganesha on a 4 node cluster and make sure all the services and ports are added properly in firewall and rpcinfo shows relevant output as below:

[root@dhcp37-180 ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad


[root@dhcp37-158 bin]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad


[root@dhcp37-127 bin]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad

[root@dhcp37-174 bin]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad

[root@dhcp37-180 ~]# firewall-cmd --list-ports
662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp

[root@dhcp37-158 statd]# firewall-cmd --list-ports
662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp

[root@dhcp37-127 statd]# firewall-cmd --list-ports
662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp

[root@dhcp37-174 statd]# firewall-cmd --list-ports
662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp


2. Now power off 3 of the nodes from the cluster and start it after 5 minutes.
3. Make sure once the nodes comes up, shared storage is mounted properly on all the nodes.
4. Start pcsd, pacemaker and nfs-ganesha service on all the 3 nodes.
5. Observe that on 2 nodes, nfs-ganesha service status shows "failed to connect to statd"

[root@dhcp37-174 ~]# service nfs-ganesha status -l
Redirecting to /bin/systemctl status  -l nfs-ganesha.service
● nfs-ganesha.service - NFS-Ganesha file server
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-04-04 08:57:29 IST; 12min ago
     Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
  Process: 3068 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS)
  Process: 3066 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS)
 Main PID: 3067 (ganesha.nfsd)
   CGroup: /system.slice/nfs-ganesha.service
           └─3067 /usr/bin/ganesha.nfsd

Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nsm_connect :NLM :CRIT :failed to connect to statd
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returnedL
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_start :NFS STARTUP :EVENT :------------------------------------------
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_start :NFS STARTUP :EVENT :------------------------------------------
Apr 04 08:58:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE



[root@dhcp37-127 nfs-ganesha]# service nfs-ganesha status -l
Redirecting to /bin/systemctl status  -l nfs-ganesha.service
● nfs-ganesha.service - NFS-Ganesha file server
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-04-04 08:55:44 IST; 14min ago
     Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
  Process: 3033 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS)
  Process: 3031 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS)
 Main PID: 3032 (ganesha.nfsd)
   CGroup: /system.slice/nfs-ganesha.service
           └─3032 /usr/bin/ganesha.nfsd

Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nsm_connect :NLM :CRIT :failed to connect to statd
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Apr 04 08:56:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE


6. rpcinfo on 2 nodes has missing entries for status while on 3rd node its proper.

[root@dhcp37-158 ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp    662  status
    100024    1   tcp    662  status
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad


[root@dhcp37-127 nfs-ganesha]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad


[root@dhcp37-174 ~]# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100003    3   udp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   udp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100021    4   udp  32000  nlockmgr
    100021    4   tcp  32000  nlockmgr
    100011    1   udp   4501  rquotad
    100011    1   tcp   4501  rquotad
    100011    2   udp   4501  rquotad
    100011    2   tcp   4501  rquotad

7. below messages are observed in /var/log/messages;


on 1st node:

Apr  4 08:55:44 dhcp37-127 rpc.statd[3029]: Version 1.3.0 starting
Apr  4 08:55:44 dhcp37-127 rpc.statd[3029]: Flags: TI-RPC
Apr  4 08:55:44 dhcp37-127 rpc.statd[3029]: Failed to open directory sm: Permission denied
Apr  4 08:55:44 dhcp37-127 rpc.statd[3029]: Failed to open /var/lib/nfs/statd/state: Permission denied
Apr  4 08:55:44 dhcp37-127 systemd: nfs-ganesha-lock.service: control process exited, code=exited status=1
Apr  4 08:55:44 dhcp37-127 systemd: Failed to start NFS status monitor for NFSv2/3 locking..
Apr  4 08:55:44 dhcp37-127 systemd: Unit nfs-ganesha-lock.service entered failed state.
Apr  4 08:55:44 dhcp37-127 systemd: nfs-ganesha-lock.service failed.
Apr  4 08:55:44 dhcp37-127 systemd: Starting NFS-Ganesha file server...


Apr  4 08:55:44 dhcp37-127 nfs-ganesha[3032]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
Apr  4 08:55:44 dhcp37-127 nfs-ganesha[3032]: [main] nsm_connect :NLM :CRIT :failed to connect to statd
Apr  4 08:55:44 dhcp37-127 nfs-ganesha[3032]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL

on 2nd node:



Apr  4 08:57:29 dhcp37-174 rpc.statd[3064]: Failed to open directory sm: Permission denied
Apr  4 08:57:29 dhcp37-174 rpc.statd[3064]: Failed to open /var/lib/nfs/statd/state: Permission denied
Apr  4 08:57:29 dhcp37-174 systemd: nfs-ganesha-lock.service: control process exited, code=exited status=1
Apr  4 08:57:29 dhcp37-174 systemd: Failed to start NFS status monitor for NFSv2/3 locking..
Apr  4 08:57:29 dhcp37-174 systemd: Unit nfs-ganesha-lock.service entered failed state.
Apr  4 08:57:29 dhcp37-174 systemd: nfs-ganesha-lock.service failed.
Apr  4 08:57:29 dhcp37-174 systemd: Starting NFS-Ganesha file server...

Apr  4 08:57:29 dhcp37-174 nfs-ganesha[3067]: [main] nsm_connect :NLM :CRIT :failed to connect to statd
Apr  4 08:57:29 dhcp37-174 nfs-ganesha[3067]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL


Actual results:

nfs-ganesha.service status shows "failed to connect to statd" after node reboot

Expected results:

there should not be any failures after node comes up and after the restart of nfs-ganesha, pcs and pacemaker services.


Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-04 11:02:57 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Shashank Raj on 2016-04-04 11:10:04 EDT ---

sosreports are placed under http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1323740

--- Additional comment from Soumya Koduri on 2016-04-04 11:21:31 EDT ---

I see below AVCs in one of the machines where rpc.statd hasn't started.

type=AVC msg=audit(1459740344.745:419): avc:  denied  { read } for  pid=3029 comm="rpc.statd" name="nfs" dev="dm-0" ino=34567184 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file
type=SYSCALL msg=audit(1459740344.745:419): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7effa7434790 a2=90800 a3=0 items=0 ppid=3028 pid=3029 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)
type=AVC msg=audit(1459740344.745:420): avc:  denied  { read } for  pid=3029 comm="rpc.statd" name="nfs" dev="dm-0" ino=34567184 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file
type=SYSCALL msg=audit(1459740344.745:420): arch=c000003e syscall=2 success=no exit=-13 a0=7effa7434750 a1=0 a2=7effa7434768 a3=5 items=0 ppid=3028 pid=3029 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)


Not sure why these AVCs are not seen on other machines.. Could you check with selinux disabled?

--- Additional comment from Shashank Raj on 2016-04-05 03:02:27 EDT ---

Correct Soumya, after running the same test with selinux disabled, i didnt observe the issue. No statd related failures seen in ganesha.service status. However i can below avc's in audit.log


type=AVC msg=audit(1459799848.045:869): avc:  denied  { read } for  pid=1565 comm="rpc.statd" name="nfs" dev="dm-0" ino=35254482 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file

type=AVC msg=audit(1459799848.045:869): avc:  denied  { read } for  pid=1565 comm="rpc.statd" name="sm" dev="fuse" ino=9851517453928257202 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=SYSCALL msg=audit(1459799848.045:869): arch=c000003e syscall=257 success=yes exit=7 a0=ffffffffffffff9c a1=7f92e6f96790 a2=90800 a3=0 items=0 ppid=1564 pid=1565 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)

type=AVC msg=audit(1459799848.060:870): avc:  denied  { read } for  pid=1565 comm="rpc.statd" name="state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1459799848.060:870): avc:  denied  { open } for  pid=1565 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1459799848.060:870): avc:  denied  { read } for  pid=1565 comm="rpc.statd" name="state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1459799848.060:870): avc:  denied  { open } for  pid=1565 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=SYSCALL msg=audit(1459799848.060:870): arch=c000003e syscall=2 success=yes exit=7 a0=7f92e6f96750 a1=0 a2=7f92e6f96768 a3=5 items=0 ppid=1564 pid=1565 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)

type=AVC msg=audit(1459799848.065:871): avc:  denied  { write } for  pid=1565 comm="rpc.statd" name="statd" dev="fuse" ino=9574569421130904447 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1459799848.065:871): avc:  denied  { add_name } for  pid=1565 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1459799848.065:871): avc:  denied  { create } for  pid=1565 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1459799848.065:871): avc:  denied  { write } for  pid=1565 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=12901113835499053102 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=SYSCALL msg=audit(1459799848.065:871): arch=c000003e syscall=2 success=yes exit=7 a0=7f92e6f96780 a1=101241 a2=1a4 a3=18 items=0 ppid=1564 pid=1565 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)

type=AVC msg=audit(1459799848.079:872): avc:  denied  { remove_name } for  pid=1565 comm="rpc.statd" name="state.new" dev="fuse" ino=12901113835499053102 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1459799848.079:872): avc:  denied  { rename } for  pid=1565 comm="rpc.statd" name="state.new" dev="fuse" ino=12901113835499053102 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1459799848.079:872): avc:  denied  { unlink } for  pid=1565 comm="rpc.statd" name="state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

Comment 6 Shashank Raj 2016-04-15 14:02:46 UTC
[root@dhcp37-180 /]# mount | grep /var/lib
[root@dhcp37-180 /]# ls -RZ /var/lib/nfs
lrwxrwxrwx. root root system_u:object_r:var_lib_t:s0   /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs

Comment 9 Shashank Raj 2016-04-28 17:22:02 UTC
As mentioned in comment 4, there are still some AVC's seen in audit logs which restrict the starting of statd in enforcing mode:

type=AVC msg=audit(1461741415.944:384): avc:  denied  { read } for  pid=1911 comm="sm-notify" name="nfs" dev="dm-0" ino=36742185 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file

type=AVC msg=audit(1461616003.027:1760): avc:  denied  { read } for  pid=18230 comm="rpc.statd" name="nfs" dev="dm-0" ino=34738912 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file

So we need to have an updated local policy which could fix the above issues as well.

Let me know for any other information

Comment 10 Lukas Vrabec 2016-04-29 09:12:50 UTC
Hi Shashank, 

AVCs with fusefs_t filesystem will be fixed by new boolean for rpcd. 

I would like to ask, how is link file "/var/lib/nfs" created? We need to ensure that link file will be created with proper SELinux label.

Comment 11 Shashank Raj 2016-04-29 10:08:28 UTC
so once we have the shared volume mounted on /var/run/gluster/shared_storage (which is a fuse mount), as part of ganesha configuration (gluster nfs-ganesha enable), we create directories under nfs-ganesha on the shared_storage location and give a link to /var/lib/nfs. As below

/var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs

/var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-158.lab.eng.blr.redhat.com/nfs

/var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-174.lab.eng.blr.redhat.com/nfs

Comment 12 Lukas Vrabec 2016-04-29 10:15:27 UTC
Understand, and which process creating link? I need to create SELinux transition rule.

Comment 13 Shashank Raj 2016-04-29 10:35:24 UTC
During gluster nfs-ganesha enable, we call this script "/usr/libexec/ganesha/ganesha-ha.sh" to configure it.

Comment 14 Lukas Vrabec 2016-05-02 08:10:34 UTC
I sent scratch builds to Shashank Raj. 

Please, attach AVCs after testing. 

Thank you.

Comment 15 Shashank Raj 2016-05-02 09:46:07 UTC
You asked me to update libselinux and policycoreutils packages, before updating selinux policies but there are other dependent packages as well which stops me from upgrading this. See below:

--> Finished Dependency Resolution
Error: Package: libselinux-2.5-3.el7.x86_64 (/libselinux-2.5-3.el7.x86_64)
           Requires: libsepol(x86-64) >= 2.5
           Installed: libsepol-2.1.9-3.el7.x86_64 (@anaconda/7.2)
               libsepol(x86-64) = 2.1.9-3.el7

Error: Package: policycoreutils-2.5-2.el7.x86_64 (/policycoreutils-2.5-2.el7.x86_64)
           Requires: libsepol >= 2.5
           Installed: libsepol-2.1.9-3.el7.x86_64 (@anaconda/7.2)
               libsepol = 2.1.9-3.el7

i downloaded the package libsepol-2.5-2.1.el7.x86_64.rpm from location (https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=489564) and then tried installing but again got into below dependency issue.

Error: libselinux conflicts with systemd-219-19.el7_2.8.x86_64

Error: Package: policycoreutils-python-2.5-2.el7.x86_64 (/policycoreutils-python-2.5-2.el7.x86_64)
           Requires: libsemanage-python >= 2.5
           Installed: libsemanage-python-2.1.10-18.el7.x86_64 (@rhel-7-server-rpms)
               libsemanage-python = 2.1.10-18.el7
           Available: libsemanage-python-2.1.10-16.el7.x86_64 (rhel-7-server-rpms)
               libsemanage-python = 2.1.10-16.el7

After downloading the libsemanage packages from (https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=489872) and try upgrading, got into below issue

Error: libsemanage conflicts with selinux-policy-targeted-3.13.1-60.el7_2.3.noarch

So, because of so many dependency issues, i could no start with the verification.

Can you please provide the required and necessary links/packages which i can use and test the fix.

Comment 16 Shashank Raj 2016-05-02 13:12:19 UTC
After upgrading to the selinux build:

[root@dhcp43-188 ~]# rpm -qa|grep selinux
selinux-policy-targeted-3.13.1-69.el7.1.noarch
selinux-policy-devel-3.13.1-69.el7.1.noarch
selinux-policy-3.13.1-69.el7.1.noarch

below AVC's are seen in audit.log

type=AVC msg=audit(1462213059.347:67507): avc:  denied  { read } for  pid=25805 comm="rpc.statd" name="nfs" dev="dm-0" ino=33768381 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file

type=AVC msg=audit(1462213059.347:67507): avc:  denied  { read } for  pid=25805 comm="rpc.statd" name="sm" dev="fuse" ino=12705707374506975531 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1462213059.358:67508): avc:  denied  { read } for  pid=25805 comm="rpc.statd" name="state" dev="fuse" ino=12079666029056761640 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.358:67508): avc:  denied  { open } for  pid=25805 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-83.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=12079666029056761640 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.166:67402): avc:  denied  { open } for  pid=32468 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-115.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=12458638062750435886 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.759:67450): avc:  denied  { open } for  pid=22099 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-133.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=11945482723556650030 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.646:62799): avc:  denied  { open } for  pid=23423 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-188.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=11315594939681067586 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.361:67509): avc:  denied  { write } for  pid=25805 comm="rpc.statd" name="statd" dev="fuse" ino=11774365275056367482 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1462213059.361:67509): avc:  denied  { add_name } for  pid=25805 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1462213059.361:67509): avc:  denied  { create } for  pid=25805 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.361:67509): avc:  denied  { write } for  pid=25805 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-83.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=10157941417756713419 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.168:67403): avc:  denied  { write } for  pid=32468 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-115.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=13750575941116322667 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.761:67451): avc:  denied  { write } for  pid=22099 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-133.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=12534429915005777352 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.648:62800): avc:  denied  { write } for  pid=23423 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-188.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=10074523560772388662 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.430:67510): avc:  denied  { remove_name } for  pid=25805 comm="rpc.statd" name="state.new" dev="fuse" ino=10157941417756713419 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir

type=AVC msg=audit(1462213059.430:67510): avc:  denied  { rename } for  pid=25805 comm="rpc.statd" name="state.new" dev="fuse" ino=10157941417756713419 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

type=AVC msg=audit(1462213059.430:67510): avc:  denied  { unlink } for  pid=25805 comm="rpc.statd" name="state" dev="fuse" ino=12079666029056761640 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file

Comment 22 James Christensen 2016-05-17 19:50:42 UTC
Fresh install today, looks like the same issue here.

-- Unit rpc-statd.service has begun starting up.
May 17 14:32:22 nfs1 rpc.statd[17967]: Version 1.3.0 starting
May 17 14:32:22 nfs1 rpc.statd[17967]: Flags: TI-RPC
May 17 14:32:22 nfs1 rpc.statd[17967]: Failed to open directory sm: Permission denied
May 17 14:32:22 nfs1 rpc.statd[17967]: Failed to open /var/lib/nfs/statd/state: Permission denied
May 17 14:32:22 nfs1 systemd[1]: rpc-statd.service: control process exited, code=exited status=1
May 17 14:32:22 nfs1 systemd[1]: Failed to start NFS status monitor for NFSv2/3 locking..
-- Subject: Unit rpc-statd.service has failed

type=SERVICE_START msg=audit(1463513555.587:433): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpc-statd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=AVC msg=audit(1463513901.613:434): avc:  denied  { read } for  pid=19924 comm="rpc.statd" name="nfs" dev="dm-0" ino=34264089 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file
type=SYSCALL msg=audit(1463513901.613:434): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7fc977d5fc70 a2=90800 a3=0 items=0 ppid=19923 pid=19924 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)
type=AVC msg=audit(1463513901.613:435): avc:  denied  { read } for  pid=19924 comm="rpc.statd" name="nfs" dev="dm-0" ino=34264089 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file
type=SYSCALL msg=audit(1463513901.613:435): arch=c000003e syscall=2 success=no exit=-13 a0=7fc977d5e910 a1=0 a2=7fc977d5e928 a3=5 items=0 ppid=19923 pid=19924 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null)
type=SERVICE_START msg=audit(1463513901.616:436): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpc-statd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

Comment 23 Milos Malik 2016-05-18 14:08:44 UTC
Which selinux-policy version did you use?

# rpm -qa selinux-policy\*

Comment 24 James Christensen 2016-05-23 19:19:52 UTC
selinux-policy-3.13.1-60.el7_2.3.noarch
selinux-policy-targeted-3.13.1-60.el7_2.3.noarch

Comment 25 Milos Malik 2016-05-24 07:06:23 UTC
The fix is not present in version 3.13.1-60.el7_2.3 of selinux-policy packages, but it is present in version 3.13.1-60.el7_2.4 and above, which will be released soon.

Comment 27 errata-xmlrpc 2016-11-04 02:46:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2283.html