Description of problem: ***************************** After configuring gnfs with CTDB and testing failover scenario with disabling kernel nfs-module, the failover doesn't work because of SELinux AVC's as mentioned below. With SELinux in permissive mode , everything works fine. type=AVC msg=audit(1481108411.013:147585): avc: denied { write } for pid=696 comm="exportfs" name="nfs" dev="dm-0" ino=1718008 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=dir type=SYSCALL msg=audit(1481108411.013:147585): arch=c000003e syscall=2 success=no exit=-13 a0=7fb978518b82 a1=40 a2=180 a3=8 items=0 ppid=694 pid=696 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="exportfs" exe="/usr/sbin/exportfs" subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(1481108411.110:147586): avc: denied { name_connect } for pid=738 comm="rpcinfo" dest=2049 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket type=SYSCALL msg=audit(1481108411.110:147586): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=7fcd7b1fda90 a2=10 a3=7ffe4a1ab1a0 items=0 ppid=737 pid=738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpcinfo" exe="/usr/sbin/rpcinfo" subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(1481108411.008:147584): avc: denied { read } for pid=694 comm="nfs-linux-kerne" name="etab" dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file type=SYSCALL msg=audit(1481108411.008:147584): arch=c000003e syscall=269 success=no exit=-13 a0=ffffffffffffff9c a1=19fb8a0 a2=4 a3=7ffc42db1550 items=0 ppid=692 pid=694 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nfs-linux-kerne" exe="/usr/bin/bash" subj=system_u:system_r:ctdbd_t:s0 key=(null) Errors in CTDB logs: 2016/12/07 16:37:09.276523 [18972]: 60.nfs: exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied) NFS_HOSTNAME is not configured. statd-callout failed 2016/12/07 16:37:09.383058 [18972]: 60.nfs: ERROR: nfs failed RPC check: 2016/12/07 16:37:09.383152 [18972]: 60.nfs: rpcinfo: RPC: Remote system error - Permission denied 2016/12/07 16:37:24.763739 [18972]: 60.nfs: exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied) NFS_HOSTNAME is not configured. statd-callout failed 2016/12/07 16:37:24.867362 [18972]: 60.nfs: ERROR: nfs failed RPC check: 2016/12/07 16:37:24.867466 [18972]: 60.nfs: rpcinfo: RPC: Remote system error - Permission denied 2016/12/07 16:37:40.254511 [18972]: 60.nfs: exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied) NFS_HOSTNAME is not configured. statd-callout failed Version-Release number of selected component (if applicable): ctdb-4.4.6-2.el7rhgs.x86_64 glusterfs-3.8.4-5.el7rhgs.x86_64 selinux-policy-3.13.1-102.el7_3.4.noarch How reproducible: Always Steps to Reproduce: 1.Setup CTDB with gnfs as mentioned in admin guide 2.kill gnfs on one node , to verify the ctdb failover 3.Check the failover and audit logs Actual results: CTDB failover fails and lots of SELinux AVC's seen. Expected results: CTDB failover should succeed and there should not be any AVC. Additional info: All works fine with SELinux in permissive mode
Following workaround is based on denials from enforcing mode: # cat bz1402360.te policy_module(bz1402360,1.0) require { type ctdbd_t; type var_lib_nfs_t; type nfs_port_t; class dir { write }; class file { getattr open read }; class tcp_socket { name_connect }; } allow ctdbd_t var_lib_nfs_t : dir { write }; allow ctdbd_t var_lib_nfs_t : file { getattr open read }; allow ctdbd_t nfs_port_t : tcp_socket { name_connect }; # make -f /usr/share/selinux/devel/Makefile Compiling targeted bz1402360 module /usr/bin/checkmodule: loading policy configuration from tmp/bz1402360.tmp /usr/bin/checkmodule: policy configuration loaded /usr/bin/checkmodule: writing binary representation (version 17) to tmp/bz1402360.mod Creating targeted bz1402360.pp policy package rm tmp/bz1402360.mod.fc tmp/bz1402360.mod # semodule -i bz1402360.pp #
After setting to permissive mode, the functionality worked fine and following AVC's are seen: type=SYSCALL msg=audit(12/07/2016 20:20:34.360:152606) : arch=x86_64 syscall=faccessat success=yes exit=0 a0=0xffffffffffffff9c a1=0xef28a0 a2=R_OK a3=0x7ffd1ae8a7d0 items=0 ppid=32244 pid=32246 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:20:34.360:152606) : avc: denied { read } for pid=32246 comm=nfs-linux-kerne name=etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 20:20:34.360:152607) : arch=x86_64 syscall=stat success=yes exit=0 a0=0xef2580 a1=0x7ffd1ae8aba0 a2=0x7ffd1ae8aba0 a3=0xef2580 items=0 ppid=32244 pid=32246 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:20:34.360:152607) : avc: denied { getattr } for pid=32246 comm=nfs-linux-kerne path=/var/lib/nfs/etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 20:20:34.456:152608) : arch=x86_64 syscall=connect success=no exit=ECONNREFUSED(Connection refused) a0=0x3 a1=0x7f803052da90 a2=0x10 a3=0x7ffdfff05700 items=0 ppid=32279 pid=32280 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:20:34.456:152608) : avc: denied { name_connect } for pid=32280 comm=rpcinfo dest=2049 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket ---- type=USER_AVC msg=audit(12/07/2016 20:21:01.681:152614) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc: received setenforce notice (enforcing=0) exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?' ---- type=SYSCALL msg=audit(12/07/2016 20:21:05.345:152619) : arch=x86_64 syscall=faccessat success=yes exit=0 a0=0xffffffffffffff9c a1=0xb0f8a0 a2=R_OK a3=0x7ffd5793e920 items=0 ppid=32698 pid=32700 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:21:05.345:152619) : avc: denied { read } for pid=32700 comm=nfs-linux-kerne name=etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 20:21:05.346:152620) : arch=x86_64 syscall=stat success=yes exit=0 a0=0xb0f580 a1=0x7ffd5793ecf0 a2=0x7ffd5793ecf0 a3=0xb0f580 items=0 ppid=32698 pid=32700 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:21:05.346:152620) : avc: denied { getattr } for pid=32700 comm=nfs-linux-kerne path=/var/lib/nfs/etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 20:21:05.432:152621) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7ff4fa644a90 a2=0x10 a3=0x7ffc728f5780 items=0 ppid=32731 pid=32732 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:21:05.432:152621) : avc: denied { name_connect } for pid=32732 comm=rpcinfo dest=2049 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket ---- type=SYSCALL msg=audit(12/07/2016 20:21:05.445:152622) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7fb5bb0d1a90 a2=0x10 a3=0x7ffd1623bc90 items=0 ppid=32735 pid=32736 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 20:21:05.445:152622) : avc: denied { name_connect } for pid=32736 comm=rpcinfo dest=38468 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:gluster_port_t:s0 tclass=tcp_socket
After applying local policy in comment 2 I the functionality still doesn't seems to work and there are more AVC's. type=AVC msg=audit(12/07/2016 21:31:03.384:153942) : avc: denied { getattr } for pid=15968 comm=pidof path=/usr/lib/polkit-1/polkitd dev="dm-0" ino=17603828 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:policykit_exec_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 21:31:03.387:153943) : arch=x86_64 syscall=access success=yes exit=0 a0=0x2642f30 a1=X_OK a2=0x7ffd197c5e70 a3=0x11 items=0 ppid=15969 pid=15970 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=60.nfs exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 21:31:03.387:153943) : avc: denied { execute } for pid=15970 comm=60.nfs name=rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 21:31:03.387:153944) : arch=x86_64 syscall=access success=yes exit=0 a0=0x2642f30 a1=R_OK a2=0x7ffd197c5e70 a3=0x11 items=0 ppid=15969 pid=15970 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=60.nfs exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 21:31:03.387:153944) : avc: denied { read } for pid=15970 comm=60.nfs name=rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 21:31:03.387:153945) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x2642f30 a1=0x2643b30 a2=0x263d990 a3=0x7ffd197c5de0 items=0 ppid=15970 pid=15972 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpc.rquotad exe=/usr/sbin/rpc.rquotad subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 21:31:03.387:153945) : avc: denied { execute_no_trans } for pid=15972 comm=60.nfs path=/usr/sbin/rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file type=AVC msg=audit(12/07/2016 21:31:03.387:153945) : avc: denied { open } for pid=15972 comm=60.nfs path=/usr/sbin/rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file ---- type=SYSCALL msg=audit(12/07/2016 21:32:06.239:153953) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7f96048c1a90 a2=0x10 a3=0x7fff90dc0320 items=0 ppid=16811 pid=16812 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) type=AVC msg=audit(12/07/2016 21:32:06.239:153953) : avc: denied { name_connect } for pid=16812 comm=rpcinfo dest=38468 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:gluster_port_t:s0 tclass=tcp_socket In permissive mode everything works fine.
Here is the improved version of the workaround policy: # cat bz1402360.te policy_module(bz1402360,1.0) require { type ctdbd_t; type var_lib_nfs_t; type nfs_port_t; type gluster_port_t; type policykit_exec_t; type rpcd_exec_t; type rpcd_t; class dir { write }; class file { getattr open read execute }; class tcp_socket { name_connect }; class process { transition }; } allow ctdbd_t var_lib_nfs_t : dir { write }; allow ctdbd_t var_lib_nfs_t : file { getattr open read }; allow ctdbd_t nfs_port_t : tcp_socket { name_connect }; allow ctdbd_t gluster_port_t : tcp_socket { name_connect }; allow ctdbd_t policykit_exec_t : file { getattr }; allow ctdbd_t rpcd_exec_t : file { getattr open read execute }; type_transition ctdbd_t rpcd_exec_t : process rpcd_t; allow ctdbd_t rpcd_t : process { transition }; # make -f /usr/share/selinux/devel/Makefile Compiling targeted bz1402360 module /usr/bin/checkmodule: loading policy configuration from tmp/bz1402360.tmp /usr/bin/checkmodule: policy configuration loaded /usr/bin/checkmodule: writing binary representation (version 17) to tmp/bz1402360.mod Creating targeted bz1402360.pp policy package rm tmp/bz1402360.mod.fc tmp/bz1402360.mod # semodule -i bz1402360.pp #
With Above Policy The issue is not seen and failover works fine without any AVC's.
With the build mentioned above : selinux-policy-3.13.1-102.el7_3.11 , the failover scenario passes and the AVC's mentioned in BZ description is not seen. However there are other AVC's observed without any functional impact and a new medium priority BZ has been reported to track the AVC's. https://bugzilla.redhat.com/show_bug.cgi?id=1411215 Marking this BZ verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html