Hide Forgot
Created attachment 1143678 [details] audit logs +++ This bug was initially created as a clone of Bug #1323740 +++ Description of problem: nfs-ganesha.service status shows "failed to connect to statd" after node reboot Version-Release number of selected component (if applicable): glusterfs-3.7.9-1 How reproducible: twice Steps to Reproduce: 1.Configure nfs-ganesha on a 4 node cluster and make sure all the services and ports are added properly in firewall and rpcinfo shows relevant output as below: [root@dhcp37-180 ~]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad [root@dhcp37-158 bin]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad [root@dhcp37-127 bin]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad [root@dhcp37-174 bin]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad [root@dhcp37-180 ~]# firewall-cmd --list-ports 662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp [root@dhcp37-158 statd]# firewall-cmd --list-ports 662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp [root@dhcp37-127 statd]# firewall-cmd --list-ports 662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp [root@dhcp37-174 statd]# firewall-cmd --list-ports 662/udp 662/tcp 4501/udp 20048/udp 32000/udp 32000/tcp 20048/tcp 4501/tcp 2. Now power off 3 of the nodes from the cluster and start it after 5 minutes. 3. Make sure once the nodes comes up, shared storage is mounted properly on all the nodes. 4. Start pcsd, pacemaker and nfs-ganesha service on all the 3 nodes. 5. Observe that on 2 nodes, nfs-ganesha service status shows "failed to connect to statd" [root@dhcp37-174 ~]# service nfs-ganesha status -l Redirecting to /bin/systemctl status -l nfs-ganesha.service ● nfs-ganesha.service - NFS-Ganesha file server Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2016-04-04 08:57:29 IST; 12min ago Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki Process: 3068 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS) Process: 3066 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS) Main PID: 3067 (ganesha.nfsd) CGroup: /system.slice/nfs-ganesha.service └─3067 /usr/bin/ganesha.nfsd Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nsm_connect :NLM :CRIT :failed to connect to statd Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returnedL Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_start :NFS STARTUP :EVENT :------------------------------------------ Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED Apr 04 08:57:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [main] nfs_start :NFS STARTUP :EVENT :------------------------------------------ Apr 04 08:58:29 dhcp37-174.lab.eng.blr.redhat.com nfs-ganesha[3067]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE [root@dhcp37-127 nfs-ganesha]# service nfs-ganesha status -l Redirecting to /bin/systemctl status -l nfs-ganesha.service ● nfs-ganesha.service - NFS-Ganesha file server Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled) Active: active (running) since Mon 2016-04-04 08:55:44 IST; 14min ago Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki Process: 3033 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS) Process: 3031 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS) Main PID: 3032 (ganesha.nfsd) CGroup: /system.slice/nfs-ganesha.service └─3032 /usr/bin/ganesha.nfsd Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nsm_connect :NLM :CRIT :failed to connect to statd Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED Apr 04 08:55:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- Apr 04 08:56:44 dhcp37-127.lab.eng.blr.redhat.com nfs-ganesha[3032]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE 6. rpcinfo on 2 nodes has missing entries for status while on 3rd node its proper. [root@dhcp37-158 ~]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 662 status 100024 1 tcp 662 status 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad [root@dhcp37-127 nfs-ganesha]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad [root@dhcp37-174 ~]# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100003 3 udp 2049 nfs 100003 3 tcp 2049 nfs 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs 100005 1 udp 20048 mountd 100005 1 tcp 20048 mountd 100005 3 udp 20048 mountd 100005 3 tcp 20048 mountd 100021 4 udp 32000 nlockmgr 100021 4 tcp 32000 nlockmgr 100011 1 udp 4501 rquotad 100011 1 tcp 4501 rquotad 100011 2 udp 4501 rquotad 100011 2 tcp 4501 rquotad 7. below messages are observed in /var/log/messages; on 1st node: Apr 4 08:55:44 dhcp37-127 rpc.statd[3029]: Version 1.3.0 starting Apr 4 08:55:44 dhcp37-127 rpc.statd[3029]: Flags: TI-RPC Apr 4 08:55:44 dhcp37-127 rpc.statd[3029]: Failed to open directory sm: Permission denied Apr 4 08:55:44 dhcp37-127 rpc.statd[3029]: Failed to open /var/lib/nfs/statd/state: Permission denied Apr 4 08:55:44 dhcp37-127 systemd: nfs-ganesha-lock.service: control process exited, code=exited status=1 Apr 4 08:55:44 dhcp37-127 systemd: Failed to start NFS status monitor for NFSv2/3 locking.. Apr 4 08:55:44 dhcp37-127 systemd: Unit nfs-ganesha-lock.service entered failed state. Apr 4 08:55:44 dhcp37-127 systemd: nfs-ganesha-lock.service failed. Apr 4 08:55:44 dhcp37-127 systemd: Starting NFS-Ganesha file server... Apr 4 08:55:44 dhcp37-127 nfs-ganesha[3032]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE Apr 4 08:55:44 dhcp37-127 nfs-ganesha[3032]: [main] nsm_connect :NLM :CRIT :failed to connect to statd Apr 4 08:55:44 dhcp37-127 nfs-ganesha[3032]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL on 2nd node: Apr 4 08:57:29 dhcp37-174 rpc.statd[3064]: Failed to open directory sm: Permission denied Apr 4 08:57:29 dhcp37-174 rpc.statd[3064]: Failed to open /var/lib/nfs/statd/state: Permission denied Apr 4 08:57:29 dhcp37-174 systemd: nfs-ganesha-lock.service: control process exited, code=exited status=1 Apr 4 08:57:29 dhcp37-174 systemd: Failed to start NFS status monitor for NFSv2/3 locking.. Apr 4 08:57:29 dhcp37-174 systemd: Unit nfs-ganesha-lock.service entered failed state. Apr 4 08:57:29 dhcp37-174 systemd: nfs-ganesha-lock.service failed. Apr 4 08:57:29 dhcp37-174 systemd: Starting NFS-Ganesha file server... Apr 4 08:57:29 dhcp37-174 nfs-ganesha[3067]: [main] nsm_connect :NLM :CRIT :failed to connect to statd Apr 4 08:57:29 dhcp37-174 nfs-ganesha[3067]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL Actual results: nfs-ganesha.service status shows "failed to connect to statd" after node reboot Expected results: there should not be any failures after node comes up and after the restart of nfs-ganesha, pcs and pacemaker services. Additional info: --- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-04 11:02:57 EDT --- This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. If this bug should be proposed for a different release, please manually change the proposed release flag. --- Additional comment from Shashank Raj on 2016-04-04 11:10:04 EDT --- sosreports are placed under http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1323740 --- Additional comment from Soumya Koduri on 2016-04-04 11:21:31 EDT --- I see below AVCs in one of the machines where rpc.statd hasn't started. type=AVC msg=audit(1459740344.745:419): avc: denied { read } for pid=3029 comm="rpc.statd" name="nfs" dev="dm-0" ino=34567184 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=SYSCALL msg=audit(1459740344.745:419): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7effa7434790 a2=90800 a3=0 items=0 ppid=3028 pid=3029 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) type=AVC msg=audit(1459740344.745:420): avc: denied { read } for pid=3029 comm="rpc.statd" name="nfs" dev="dm-0" ino=34567184 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=SYSCALL msg=audit(1459740344.745:420): arch=c000003e syscall=2 success=no exit=-13 a0=7effa7434750 a1=0 a2=7effa7434768 a3=5 items=0 ppid=3028 pid=3029 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) Not sure why these AVCs are not seen on other machines.. Could you check with selinux disabled? --- Additional comment from Shashank Raj on 2016-04-05 03:02:27 EDT --- Correct Soumya, after running the same test with selinux disabled, i didnt observe the issue. No statd related failures seen in ganesha.service status. However i can below avc's in audit.log type=AVC msg=audit(1459799848.045:869): avc: denied { read } for pid=1565 comm="rpc.statd" name="nfs" dev="dm-0" ino=35254482 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=AVC msg=audit(1459799848.045:869): avc: denied { read } for pid=1565 comm="rpc.statd" name="sm" dev="fuse" ino=9851517453928257202 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=SYSCALL msg=audit(1459799848.045:869): arch=c000003e syscall=257 success=yes exit=7 a0=ffffffffffffff9c a1=7f92e6f96790 a2=90800 a3=0 items=0 ppid=1564 pid=1565 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) type=AVC msg=audit(1459799848.060:870): avc: denied { read } for pid=1565 comm="rpc.statd" name="state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1459799848.060:870): avc: denied { open } for pid=1565 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1459799848.060:870): avc: denied { read } for pid=1565 comm="rpc.statd" name="state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1459799848.060:870): avc: denied { open } for pid=1565 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=SYSCALL msg=audit(1459799848.060:870): arch=c000003e syscall=2 success=yes exit=7 a0=7f92e6f96750 a1=0 a2=7f92e6f96768 a3=5 items=0 ppid=1564 pid=1565 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) type=AVC msg=audit(1459799848.065:871): avc: denied { write } for pid=1565 comm="rpc.statd" name="statd" dev="fuse" ino=9574569421130904447 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1459799848.065:871): avc: denied { add_name } for pid=1565 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1459799848.065:871): avc: denied { create } for pid=1565 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1459799848.065:871): avc: denied { write } for pid=1565 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=12901113835499053102 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=SYSCALL msg=audit(1459799848.065:871): arch=c000003e syscall=2 success=yes exit=7 a0=7f92e6f96780 a1=101241 a2=1a4 a3=18 items=0 ppid=1564 pid=1565 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) type=AVC msg=audit(1459799848.079:872): avc: denied { remove_name } for pid=1565 comm="rpc.statd" name="state.new" dev="fuse" ino=12901113835499053102 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1459799848.079:872): avc: denied { rename } for pid=1565 comm="rpc.statd" name="state.new" dev="fuse" ino=12901113835499053102 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1459799848.079:872): avc: denied { unlink } for pid=1565 comm="rpc.statd" name="state" dev="fuse" ino=13562855280619438618 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file
[root@dhcp37-180 /]# mount | grep /var/lib [root@dhcp37-180 /]# ls -RZ /var/lib/nfs lrwxrwxrwx. root root system_u:object_r:var_lib_t:s0 /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs
As mentioned in comment 4, there are still some AVC's seen in audit logs which restrict the starting of statd in enforcing mode: type=AVC msg=audit(1461741415.944:384): avc: denied { read } for pid=1911 comm="sm-notify" name="nfs" dev="dm-0" ino=36742185 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=AVC msg=audit(1461616003.027:1760): avc: denied { read } for pid=18230 comm="rpc.statd" name="nfs" dev="dm-0" ino=34738912 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file So we need to have an updated local policy which could fix the above issues as well. Let me know for any other information
Hi Shashank, AVCs with fusefs_t filesystem will be fixed by new boolean for rpcd. I would like to ask, how is link file "/var/lib/nfs" created? We need to ensure that link file will be created with proper SELinux label.
so once we have the shared volume mounted on /var/run/gluster/shared_storage (which is a fuse mount), as part of ganesha configuration (gluster nfs-ganesha enable), we create directories under nfs-ganesha on the shared_storage location and give a link to /var/lib/nfs. As below /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-180.lab.eng.blr.redhat.com/nfs /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-158.lab.eng.blr.redhat.com/nfs /var/lib/nfs -> /var/run/gluster/shared_storage/nfs-ganesha/dhcp37-174.lab.eng.blr.redhat.com/nfs
Understand, and which process creating link? I need to create SELinux transition rule.
During gluster nfs-ganesha enable, we call this script "/usr/libexec/ganesha/ganesha-ha.sh" to configure it.
I sent scratch builds to Shashank Raj. Please, attach AVCs after testing. Thank you.
You asked me to update libselinux and policycoreutils packages, before updating selinux policies but there are other dependent packages as well which stops me from upgrading this. See below: --> Finished Dependency Resolution Error: Package: libselinux-2.5-3.el7.x86_64 (/libselinux-2.5-3.el7.x86_64) Requires: libsepol(x86-64) >= 2.5 Installed: libsepol-2.1.9-3.el7.x86_64 (@anaconda/7.2) libsepol(x86-64) = 2.1.9-3.el7 Error: Package: policycoreutils-2.5-2.el7.x86_64 (/policycoreutils-2.5-2.el7.x86_64) Requires: libsepol >= 2.5 Installed: libsepol-2.1.9-3.el7.x86_64 (@anaconda/7.2) libsepol = 2.1.9-3.el7 i downloaded the package libsepol-2.5-2.1.el7.x86_64.rpm from location (https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=489564) and then tried installing but again got into below dependency issue. Error: libselinux conflicts with systemd-219-19.el7_2.8.x86_64 Error: Package: policycoreutils-python-2.5-2.el7.x86_64 (/policycoreutils-python-2.5-2.el7.x86_64) Requires: libsemanage-python >= 2.5 Installed: libsemanage-python-2.1.10-18.el7.x86_64 (@rhel-7-server-rpms) libsemanage-python = 2.1.10-18.el7 Available: libsemanage-python-2.1.10-16.el7.x86_64 (rhel-7-server-rpms) libsemanage-python = 2.1.10-16.el7 After downloading the libsemanage packages from (https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=489872) and try upgrading, got into below issue Error: libsemanage conflicts with selinux-policy-targeted-3.13.1-60.el7_2.3.noarch So, because of so many dependency issues, i could no start with the verification. Can you please provide the required and necessary links/packages which i can use and test the fix.
After upgrading to the selinux build: [root@dhcp43-188 ~]# rpm -qa|grep selinux selinux-policy-targeted-3.13.1-69.el7.1.noarch selinux-policy-devel-3.13.1-69.el7.1.noarch selinux-policy-3.13.1-69.el7.1.noarch below AVC's are seen in audit.log type=AVC msg=audit(1462213059.347:67507): avc: denied { read } for pid=25805 comm="rpc.statd" name="nfs" dev="dm-0" ino=33768381 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=AVC msg=audit(1462213059.347:67507): avc: denied { read } for pid=25805 comm="rpc.statd" name="sm" dev="fuse" ino=12705707374506975531 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1462213059.358:67508): avc: denied { read } for pid=25805 comm="rpc.statd" name="state" dev="fuse" ino=12079666029056761640 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.358:67508): avc: denied { open } for pid=25805 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-83.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=12079666029056761640 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.166:67402): avc: denied { open } for pid=32468 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-115.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=12458638062750435886 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.759:67450): avc: denied { open } for pid=22099 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-133.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=11945482723556650030 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.646:62799): avc: denied { open } for pid=23423 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-188.lab.eng.blr.redhat.com/nfs/statd/state" dev="fuse" ino=11315594939681067586 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.361:67509): avc: denied { write } for pid=25805 comm="rpc.statd" name="statd" dev="fuse" ino=11774365275056367482 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1462213059.361:67509): avc: denied { add_name } for pid=25805 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1462213059.361:67509): avc: denied { create } for pid=25805 comm="rpc.statd" name="state.new" scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.361:67509): avc: denied { write } for pid=25805 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-83.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=10157941417756713419 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.168:67403): avc: denied { write } for pid=32468 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp42-115.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=13750575941116322667 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.761:67451): avc: denied { write } for pid=22099 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-133.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=12534429915005777352 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.648:62800): avc: denied { write } for pid=23423 comm="rpc.statd" path="/run/gluster/shared_storage/nfs-ganesha/dhcp43-188.lab.eng.blr.redhat.com/nfs/statd/state.new" dev="fuse" ino=10074523560772388662 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.430:67510): avc: denied { remove_name } for pid=25805 comm="rpc.statd" name="state.new" dev="fuse" ino=10157941417756713419 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=dir type=AVC msg=audit(1462213059.430:67510): avc: denied { rename } for pid=25805 comm="rpc.statd" name="state.new" dev="fuse" ino=10157941417756713419 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file type=AVC msg=audit(1462213059.430:67510): avc: denied { unlink } for pid=25805 comm="rpc.statd" name="state" dev="fuse" ino=12079666029056761640 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:fusefs_t:s0 tclass=file
Fresh install today, looks like the same issue here. -- Unit rpc-statd.service has begun starting up. May 17 14:32:22 nfs1 rpc.statd[17967]: Version 1.3.0 starting May 17 14:32:22 nfs1 rpc.statd[17967]: Flags: TI-RPC May 17 14:32:22 nfs1 rpc.statd[17967]: Failed to open directory sm: Permission denied May 17 14:32:22 nfs1 rpc.statd[17967]: Failed to open /var/lib/nfs/statd/state: Permission denied May 17 14:32:22 nfs1 systemd[1]: rpc-statd.service: control process exited, code=exited status=1 May 17 14:32:22 nfs1 systemd[1]: Failed to start NFS status monitor for NFSv2/3 locking.. -- Subject: Unit rpc-statd.service has failed type=SERVICE_START msg=audit(1463513555.587:433): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpc-statd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' type=AVC msg=audit(1463513901.613:434): avc: denied { read } for pid=19924 comm="rpc.statd" name="nfs" dev="dm-0" ino=34264089 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=SYSCALL msg=audit(1463513901.613:434): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7fc977d5fc70 a2=90800 a3=0 items=0 ppid=19923 pid=19924 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) type=AVC msg=audit(1463513901.613:435): avc: denied { read } for pid=19924 comm="rpc.statd" name="nfs" dev="dm-0" ino=34264089 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=lnk_file type=SYSCALL msg=audit(1463513901.613:435): arch=c000003e syscall=2 success=no exit=-13 a0=7fc977d5e910 a1=0 a2=7fc977d5e928 a3=5 items=0 ppid=19923 pid=19924 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc.statd" exe="/usr/sbin/rpc.statd" subj=system_u:system_r:rpcd_t:s0 key=(null) type=SERVICE_START msg=audit(1463513901.616:436): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpc-statd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Which selinux-policy version did you use? # rpm -qa selinux-policy\*
selinux-policy-3.13.1-60.el7_2.3.noarch selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
The fix is not present in version 3.13.1-60.el7_2.3 of selinux-policy packages, but it is present in version 3.13.1-60.el7_2.4 and above, which will be released soon.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2283.html