Bug 1402360 - CTDB:NFS: CTDB failover doesn't work because of SELinux AVC's
Summary: CTDB:NFS: CTDB failover doesn't work because of SELinux AVC's
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gluster-nfs
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.2.0
Assignee: Soumya Koduri
QA Contact: surabhi
URL:
Whiteboard:
Depends On: 1403266
Blocks: 1351528
TreeView+ depends on / blocked
 
Reported: 2016-12-07 11:09 UTC by surabhi
Modified: 2017-03-23 05:55 UTC (History)
7 users (show)

Fixed In Version: selinux-policy-3.13.1-102.el7_3.11
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1402451 (view as bug list)
Environment:
Last Closed: 2017-03-23 05:55:00 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description surabhi 2016-12-07 11:09:00 UTC
Description of problem:

*****************************
After configuring gnfs with CTDB and testing failover scenario with disabling kernel nfs-module, the failover doesn't work because of SELinux AVC's as mentioned below.

With SELinux in permissive mode , everything works fine.

type=AVC msg=audit(1481108411.013:147585): avc:  denied  { write } for  pid=696 comm="exportfs" name="nfs" dev="dm-0" ino=1718008 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=dir
type=SYSCALL msg=audit(1481108411.013:147585): arch=c000003e syscall=2 success=no exit=-13 a0=7fb978518b82 a1=40 a2=180 a3=8 items=0 ppid=694 pid=696 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="exportfs" exe="/usr/sbin/exportfs" subj=system_u:system_r:ctdbd_t:s0 key=(null)
type=AVC msg=audit(1481108411.110:147586): avc:  denied  { name_connect } for  pid=738 comm="rpcinfo" dest=2049 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1481108411.110:147586): arch=c000003e syscall=42 success=no exit=-13 a0=3 a1=7fcd7b1fda90 a2=10 a3=7ffe4a1ab1a0 items=0 ppid=737 pid=738 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpcinfo" exe="/usr/sbin/rpcinfo" subj=system_u:system_r:ctdbd_t:s0 key=(null)
type=AVC msg=audit(1481108411.008:147584): avc:  denied  { read } for  pid=694 comm="nfs-linux-kerne" name="etab" dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file
type=SYSCALL msg=audit(1481108411.008:147584): arch=c000003e syscall=269 success=no exit=-13 a0=ffffffffffffff9c a1=19fb8a0 a2=4 a3=7ffc42db1550 items=0 ppid=692 pid=694 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nfs-linux-kerne" exe="/usr/bin/bash" subj=system_u:system_r:ctdbd_t:s0 key=(null)


Errors in CTDB logs:

2016/12/07 16:37:09.276523 [18972]: 60.nfs: exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied)
NFS_HOSTNAME is not configured. statd-callout failed
2016/12/07 16:37:09.383058 [18972]: 60.nfs: ERROR: nfs failed RPC check:
2016/12/07 16:37:09.383152 [18972]: 60.nfs: rpcinfo: RPC: Remote system error - Permission denied
2016/12/07 16:37:24.763739 [18972]: 60.nfs: exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied)
NFS_HOSTNAME is not configured. statd-callout failed
2016/12/07 16:37:24.867362 [18972]: 60.nfs: ERROR: nfs failed RPC check:
2016/12/07 16:37:24.867466 [18972]: 60.nfs: rpcinfo: RPC: Remote system error - Permission denied
2016/12/07 16:37:40.254511 [18972]: 60.nfs: exportfs: could not open /var/lib/nfs/.etab.lock for locking: errno 13 (Permission denied)
NFS_HOSTNAME is not configured. statd-callout failed



Version-Release number of selected component (if applicable):
ctdb-4.4.6-2.el7rhgs.x86_64
glusterfs-3.8.4-5.el7rhgs.x86_64
selinux-policy-3.13.1-102.el7_3.4.noarch

How reproducible:
Always

Steps to Reproduce:
1.Setup CTDB with gnfs as mentioned in admin guide
2.kill gnfs on one node , to verify the ctdb failover
3.Check the failover and audit logs

Actual results:
CTDB failover fails and lots of SELinux AVC's seen.

Expected results:
CTDB failover should succeed and there should not be any AVC.


Additional info:

All works fine with SELinux in permissive mode

Comment 2 Milos Malik 2016-12-07 11:45:58 UTC
Following workaround is based on denials from enforcing mode:

# cat bz1402360.te 
policy_module(bz1402360,1.0)

require {
  type ctdbd_t;
  type var_lib_nfs_t;
  type nfs_port_t;
  class dir { write };
  class file { getattr open read };
  class tcp_socket { name_connect };
}

allow ctdbd_t var_lib_nfs_t : dir { write };
allow ctdbd_t var_lib_nfs_t : file { getattr open read };
allow ctdbd_t nfs_port_t : tcp_socket { name_connect };

# make -f /usr/share/selinux/devel/Makefile 
Compiling targeted bz1402360 module
/usr/bin/checkmodule:  loading policy configuration from tmp/bz1402360.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/bz1402360.mod
Creating targeted bz1402360.pp policy package
rm tmp/bz1402360.mod.fc tmp/bz1402360.mod
# semodule -i bz1402360.pp
#

Comment 3 surabhi 2016-12-07 14:55:35 UTC
After setting to permissive mode, the functionality worked fine and following AVC's are seen:

type=SYSCALL msg=audit(12/07/2016 20:20:34.360:152606) : arch=x86_64 syscall=faccessat success=yes exit=0 a0=0xffffffffffffff9c a1=0xef28a0 a2=R_OK a3=0x7ffd1ae8a7d0 items=0 ppid=32244 pid=32246 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:20:34.360:152606) : avc:  denied  { read } for  pid=32246 comm=nfs-linux-kerne name=etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 20:20:34.360:152607) : arch=x86_64 syscall=stat success=yes exit=0 a0=0xef2580 a1=0x7ffd1ae8aba0 a2=0x7ffd1ae8aba0 a3=0xef2580 items=0 ppid=32244 pid=32246 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:20:34.360:152607) : avc:  denied  { getattr } for  pid=32246 comm=nfs-linux-kerne path=/var/lib/nfs/etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 20:20:34.456:152608) : arch=x86_64 syscall=connect success=no exit=ECONNREFUSED(Connection refused) a0=0x3 a1=0x7f803052da90 a2=0x10 a3=0x7ffdfff05700 items=0 ppid=32279 pid=32280 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:20:34.456:152608) : avc:  denied  { name_connect } for  pid=32280 comm=rpcinfo dest=2049 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket 
----
type=USER_AVC msg=audit(12/07/2016 20:21:01.681:152614) : pid=1 uid=root auid=unset ses=unset subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=0)  exe=/usr/lib/systemd/systemd sauid=root hostname=? addr=? terminal=?' 
----
type=SYSCALL msg=audit(12/07/2016 20:21:05.345:152619) : arch=x86_64 syscall=faccessat success=yes exit=0 a0=0xffffffffffffff9c a1=0xb0f8a0 a2=R_OK a3=0x7ffd5793e920 items=0 ppid=32698 pid=32700 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:21:05.345:152619) : avc:  denied  { read } for  pid=32700 comm=nfs-linux-kerne name=etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 20:21:05.346:152620) : arch=x86_64 syscall=stat success=yes exit=0 a0=0xb0f580 a1=0x7ffd5793ecf0 a2=0x7ffd5793ecf0 a3=0xb0f580 items=0 ppid=32698 pid=32700 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=nfs-linux-kerne exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:21:05.346:152620) : avc:  denied  { getattr } for  pid=32700 comm=nfs-linux-kerne path=/var/lib/nfs/etab dev="dm-0" ino=1718006 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:var_lib_nfs_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 20:21:05.432:152621) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7ff4fa644a90 a2=0x10 a3=0x7ffc728f5780 items=0 ppid=32731 pid=32732 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:21:05.432:152621) : avc:  denied  { name_connect } for  pid=32732 comm=rpcinfo dest=2049 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:nfs_port_t:s0 tclass=tcp_socket 
----
type=SYSCALL msg=audit(12/07/2016 20:21:05.445:152622) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7fb5bb0d1a90 a2=0x10 a3=0x7ffd1623bc90 items=0 ppid=32735 pid=32736 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 20:21:05.445:152622) : avc:  denied  { name_connect } for  pid=32736 comm=rpcinfo dest=38468 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:gluster_port_t:s0 tclass=tcp_socket

Comment 4 surabhi 2016-12-07 16:06:49 UTC
After applying local policy in comment 2 I the functionality still doesn't seems to work and there are more AVC's.


type=AVC msg=audit(12/07/2016 21:31:03.384:153942) : avc:  denied  { getattr } for  pid=15968 comm=pidof path=/usr/lib/polkit-1/polkitd dev="dm-0" ino=17603828 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:policykit_exec_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 21:31:03.387:153943) : arch=x86_64 syscall=access success=yes exit=0 a0=0x2642f30 a1=X_OK a2=0x7ffd197c5e70 a3=0x11 items=0 ppid=15969 pid=15970 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=60.nfs exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 21:31:03.387:153943) : avc:  denied  { execute } for  pid=15970 comm=60.nfs name=rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 21:31:03.387:153944) : arch=x86_64 syscall=access success=yes exit=0 a0=0x2642f30 a1=R_OK a2=0x7ffd197c5e70 a3=0x11 items=0 ppid=15969 pid=15970 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=60.nfs exe=/usr/bin/bash subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 21:31:03.387:153944) : avc:  denied  { read } for  pid=15970 comm=60.nfs name=rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 21:31:03.387:153945) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x2642f30 a1=0x2643b30 a2=0x263d990 a3=0x7ffd197c5de0 items=0 ppid=15970 pid=15972 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpc.rquotad exe=/usr/sbin/rpc.rquotad subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 21:31:03.387:153945) : avc:  denied  { execute_no_trans } for  pid=15972 comm=60.nfs path=/usr/sbin/rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file 
type=AVC msg=audit(12/07/2016 21:31:03.387:153945) : avc:  denied  { open } for  pid=15972 comm=60.nfs path=/usr/sbin/rpc.rquotad dev="dm-0" ino=17660926 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:rpcd_exec_t:s0 tclass=file 
----
type=SYSCALL msg=audit(12/07/2016 21:32:06.239:153953) : arch=x86_64 syscall=connect success=yes exit=0 a0=0x3 a1=0x7f96048c1a90 a2=0x10 a3=0x7fff90dc0320 items=0 ppid=16811 pid=16812 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rpcinfo exe=/usr/sbin/rpcinfo subj=system_u:system_r:ctdbd_t:s0 key=(null) 
type=AVC msg=audit(12/07/2016 21:32:06.239:153953) : avc:  denied  { name_connect } for  pid=16812 comm=rpcinfo dest=38468 scontext=system_u:system_r:ctdbd_t:s0 tcontext=system_u:object_r:gluster_port_t:s0 tclass=tcp_socket 


In permissive mode everything works fine.

Comment 5 Milos Malik 2016-12-08 09:44:59 UTC
Here is the improved version of the workaround policy:

# cat bz1402360.te 
policy_module(bz1402360,1.0)

require {
  type ctdbd_t;
  type var_lib_nfs_t;
  type nfs_port_t;
  type gluster_port_t;
  type policykit_exec_t;
  type rpcd_exec_t;
  type rpcd_t;
  class dir { write };
  class file { getattr open read execute };
  class tcp_socket { name_connect };
  class process { transition };
}

allow ctdbd_t var_lib_nfs_t : dir { write };
allow ctdbd_t var_lib_nfs_t : file { getattr open read };
allow ctdbd_t nfs_port_t : tcp_socket { name_connect };
allow ctdbd_t gluster_port_t : tcp_socket { name_connect };
allow ctdbd_t policykit_exec_t : file { getattr };
allow ctdbd_t rpcd_exec_t : file { getattr open read execute };
type_transition ctdbd_t rpcd_exec_t : process rpcd_t;
allow ctdbd_t rpcd_t : process { transition };

# make -f /usr/share/selinux/devel/Makefile 
Compiling targeted bz1402360 module
/usr/bin/checkmodule:  loading policy configuration from tmp/bz1402360.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/bz1402360.mod
Creating targeted bz1402360.pp policy package
rm tmp/bz1402360.mod.fc tmp/bz1402360.mod
# semodule -i bz1402360.pp 
#

Comment 7 surabhi 2016-12-09 11:38:18 UTC
With Above Policy The issue is not seen and failover works fine without any AVC's.

Comment 13 surabhi 2017-01-09 06:53:40 UTC
With the build mentioned above : selinux-policy-3.13.1-102.el7_3.11 , the failover scenario passes and the AVC's mentioned in BZ description is not seen. However there are other AVC's observed without any functional impact and a new medium priority BZ has been reported to track the AVC's. 
https://bugzilla.redhat.com/show_bug.cgi?id=1411215

Marking this BZ verified.

Comment 15 errata-xmlrpc 2017-03-23 05:55:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.