Bug 1118859

Summary: Galera wsrep_sst_rsync selinux denials
Product: Red Hat OpenStack Reporter: John Eckersberg <jeckersb>
Component: openstack-selinuxAssignee: Ryan Hallisey <rhallise>
Status: CLOSED NEXTRELEASE QA Contact: Omri Hochman <ohochman>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.0 (RHEL 7)CC: ajeain, dwalsh, hbrock, lhh, mgrepl, ohochman, rhallise, rohara, sclewis, slong, yeylon
Target Milestone: rc   
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-selinux-0.5.14-3.el7ost Doc Type: Bug Fix
Doc Text:
Previously, SELinux prevented wsrep_sst_script from performing an lsof command. SELinux also prevented MariaDB from using port 4444. As a result, Galera could not join a cluster. With this update, the wsrep_sst_script is now allowed to execute lsof and relabel port 4444 to mariadb_port_t, so that MariaDB can successfully join a cluster and the lsof command succeeds.
Story Points: ---
Clone Of:
: 1122688 (view as bug list) Environment:
Last Closed: 2014-07-23 19:58:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1122688    
Attachments:
Description Flags
audit.log
none
audit.log with 'semanage dontaudit off' none

Description John Eckersberg 2014-07-11 18:17:55 UTC
Created attachment 917411 [details]
audit.log

This is from a secondary (non-boostrap) node when trying to join the cluster.  Lots of denials.

Comment 1 Ryan O'Hara 2014-07-11 18:40:22 UTC
What is happening here is that when a node joins the cluster and needs sync, the SST is called. We use wsrep_sst_rsync, although other methods exist. Currently we don't other SST methods.

This wsrep_sst_script does a number of things, like call lsof. Clearly it is being prevented from doing this. I believe this audit.log was partially captured while in permissive mode.

Comment 4 Omri Hochman 2014-07-14 15:10:57 UTC
I believe that this BZ is blocking the scenario of Staypuft HA with SELinux enabled.   As a workaround - In QE we're disabling SELinux on the "Discovered-Hosts" by changing the staypuft Provision-templates.

Comment 5 Ryan Hallisey 2014-07-14 17:46:10 UTC
These should take care of it.

domain_read_all_domains_state(mysqld_t)
files_search_pids(mysqld_t)
files_getattr_all_sockets(mysqld_t)

Comment 6 Ryan Hallisey 2014-07-14 18:12:28 UTC
Try with the new policy and if things break, attach your log from a permissive test.

Comment 8 John Eckersberg 2014-07-14 21:23:21 UTC
If I run with selinux in permissive mode, I still get this denial:

type=AVC msg=audit(1405371699.582:201): avc:  denied  { getattr } for  pid=14592 comm="lsof" path="/usr/bin/mysqld_safe" dev="dm-0" ino=17369661 scontext=system_u:system_r:mysqld_t:s0 tcontext=system_u:object_r:mysqld_safe_exec_t:s0 tclass=file

Comment 9 Ryan Hallisey 2014-07-15 13:41:49 UTC
I don't think we need to transition it is just a read.

allow mysqld_t mysqld_safe_exec_t:file getattr

Comment 10 Ryan O'Hara 2014-07-15 23:13:14 UTC
Created attachment 918272 [details]
audit.log with 'semanage dontaudit off'

I'm using this policy and galera's node cannot join a cluster when running in enforcing mode, but no AVCs are logged. Setting to permissive works, but still no AVCs logged. I used 'semanage dontaudit off' to finally get some AVCs, seen here.

Comment 11 Ryan O'Hara 2014-07-16 02:19:10 UTC
I could be wrong, but the last AVC in comment #10 seems bad. Since mysqld forks the wsrep_rsync_sst, which runs rsync, it needs access to port 4444. Is there a quick/dirty way to give mysql_t access to that port so I can see if that is indeed the problem?

Comment 12 Ryan Hallisey 2014-07-16 14:11:19 UTC
Ryan you can use this for you test:

allow mysqld_t kerberos_port_t:tcp_socket name_bind;

Comment 13 Daniel Walsh 2014-07-16 17:00:27 UTC
Ryan O'Hara is this a standard config with mariadb listening on port 4444?

grep 4444 /etc/services 
krb524          4444/tcp        nv-video        # Kerberos 5 to 4 ticket xlator
krb524          4444/udp        nv-video        # Kerberos 5 to 4 ticket xlator


A quick fix would be to execute

semanage port -m -t mysqld_port_t -p tcp 4444

Which will relabel port 4444 as a mysqld_port_t port.

If this is standard config we could either change this port permanantly to myqsqld_port_t since not many people are using KRB4 (Probably for 20 years)

Or we could allow mysqld_d to bind to kerberos_port_t

Comment 14 Ryan O'Hara 2014-07-16 18:28:20 UTC
(In reply to Daniel Walsh from comment #13)
> Ryan O'Hara is this a standard config with mariadb listening on port 4444?
> 
> grep 4444 /etc/services 
> krb524          4444/tcp        nv-video        # Kerberos 5 to 4 ticket
> xlator
> krb524          4444/udp        nv-video        # Kerberos 5 to 4 ticket
> xlator
> 
> 
> A quick fix would be to execute
> 
> semanage port -m -t mysqld_port_t -p tcp 4444
> 
> Which will relabel port 4444 as a mysqld_port_t port.
> 
> If this is standard config we could either change this port permanantly to
> myqsqld_port_t since not many people are using KRB4 (Probably for 20 years)
> 
> Or we could allow mysqld_d to bind to kerberos_port_t

Yes. It isn't actually mysqld that listens on port 4444. When a node tries to join a galera cluster, it may need to be sync'd, and we user rsync to do this. mysqld will exec wsrep_sst_rsync, which then starts an rsync daemon listening on port 4444 to receive the sync data. This port is hard-coded in wsrep_sst_rsync. So it is standard if you're using rsync for the SST method.

If we can change port 4444 to mysqld_port_t, that seems fine. I am assuming that the forked wsrep_sst_rsync process will run in a context that has permission to bind to mysqld_port_t. Can we do this in openstack-selinux such that this is isolated to openstack?

Comment 15 Ryan Hallisey 2014-07-16 18:37:54 UTC
Yes, I'll add to the newest build.  We can add it to Fedora also as long as there isn't issue.

semanage port -m -t mysqld_port_t -p tcp 4444

Comment 16 Ryan O'Hara 2014-07-17 01:02:08 UTC
I tested openstack-selinux-0.5.14-3.el7ost this evening in enforcing mode and galera nodes were able join (performing an SST with rsync). No AVCs. Setting the Fixed In Version and moving to MODIFIED.

Note to QE -- an easy way to force an SST is to bring up a bootstrap node and then remove /var/lib/mysql/grastate.dat on the node(s) that will join the cluster. This is because an SST is not always requireded when a node joins. removing this file forces the SST.