Bug 1430402 - OSP10 -> OSP11 upgrade: nova and glance processes AVC denies are seen during upgrade
Summary: OSP10 -> OSP11 upgrade: nova and glance processes AVC denies are seen during ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 11.0 (Ocata)
Assignee: Lon Hohberger
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-08 14:29 UTC by Marius Cornea
Modified: 2017-05-17 20:06 UTC (History)
12 users (show)

Fixed In Version: openstack-selinux-0.8.5-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 20:06:21 UTC
Target Upstream Version:


Attachments (Terms of Use)
audit.log (3.24 MB, text/plain)
2017-03-28 15:47 UTC, Marius Cornea
no flags Details
Patch for observed AVCs. (2.39 KB, patch)
2017-03-30 13:37 UTC, Lon Hohberger
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Marius Cornea 2017-03-08 14:29:30 UTC
Description of problem:
OSP10 -> OSP11 upgrade: nova and glance processes AVC denies are seen during upgrade on the controller nodes:

type=AVC msg=audit(1488981402.933:6311): avc:  denied  { search } for  pid=535769 comm="nova-conductor" name="my.cnf.d" dev="vda2" ino=8409767 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=dir

type=AVC msg=audit(1488981406.178:6317): avc:  denied  { search } for  pid=535880 comm="nova-api" name="my.cnf.d" dev="vda2" ino=8409767 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=dir

type=AVC msg=audit(1488981411.085:6341): avc:  denied  { search } for  pid=536057 comm="nova-scheduler" name="my.cnf.d" dev="vda2" ino=8409767 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=dir

type=AVC msg=audit(1488981919.660:6482): avc:  denied  { search } for  pid=536149 comm="nova-consoleaut" name="my.cnf.d" dev="vda2" ino=8409767 scontext=system_u:system_r:nova_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=dir

type=AVC msg=audit(1488982615.275:7224): avc:  denied  { search } for  pid=535153 comm="glance-api" name="my.cnf.d" dev="vda2" ino=8409767 scontext=system_u:system_r:glance_api_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=dir


Version-Release number of selected component (if applicable):
openstack-selinux-0.7.13-3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Upgrade OSP10 to OSP11
2. Check /var/log/audit/audit.log on controller nodes

Actual results:
A bunch of avc denials related to the openstack process show up.

Expected results:
audit.log is clean.

Additional info:
This could be an issue with the OSP11 deployment as well but I have only seen it during the upgrade workflow.

Comment 1 Lon Hohberger 2017-03-27 19:58:45 UTC
Can you please attach the whole /var/log/audit/audit.log?

Comment 2 Lon Hohberger 2017-03-27 20:05:29 UTC
Also, are there functional problems as a result (for example, is the system not usable), or is this noise?

Comment 3 Lon Hohberger 2017-03-27 21:18:30 UTC
I'm also confused as to why Nova is trying to read mysql's configuration files...

Comment 4 Marius Cornea 2017-03-28 15:47:17 UTC
Created attachment 1267031 [details]
audit.log

Attaching audit.log. 

I couldn't determine any functional issues caused by these during my tests. 

The mysql conf files seem to be set in nova.conf:

[root@overcloud-controller-0 heat-admin]# grep sql /etc/nova/nova.conf
connection=mysql+pymysql://nova_api:R4TNM9XvM3Ptg7GcK3dnnHR4W@10.0.0.15/nova_api?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo
connection=mysql+pymysql://nova:R4TNM9XvM3Ptg7GcK3dnnHR4W@10.0.0.15/nova?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo
connection=mysql+pymysql://nova_placement:R4TNM9XvM3Ptg7GcK3dnnHR4W@10.0.0.15/nova_placement?read_default_file=/etc/my.cnf.d/tripleo.cnf&read_default_group=tripleo

[root@overcloud-controller-0 heat-admin]# cat /etc/my.cnf.d/tripleo.cnf
[tripleo]
bind-address=10.0.0.23

[heat-admin@overcloud-controller-1 ~]$ cat /etc/my.cnf.d/tripleo.cnf
[tripleo]
bind-address=10.0.0.13

[root@overcloud-controller-2 heat-admin]# cat /etc/my.cnf.d/tripleo.cnf
[tripleo]
bind-address=10.0.0.22

These are the local addresses on the internal_api network.

Comment 5 Marius Cornea 2017-03-28 15:51:42 UTC
This appears to be related to https://launchpad.net/tripleo/+bug/1643487

Comment 6 Marius Cornea 2017-03-28 16:23:29 UTC
[root@overcloud-controller-2 heat-admin]# ls -lZ /etc/my.cnf.d
-rw-r--r--. root root unconfined_u:object_r:mysqld_etc_t:s0 client.cnf
-rw-r--r--. root root system_u:object_r:mysqld_etc_t:s0 galera.cnf
-rw-r--r--. root root unconfined_u:object_r:mysqld_etc_t:s0 mysql-clients.cnf
-rw-r--r--. root root unconfined_u:object_r:mysqld_etc_t:s0 server.cnf
-rw-r--r--. root root system_u:object_r:mysqld_etc_t:s0 tripleo.cnf

Comment 7 Lon Hohberger 2017-03-28 21:18:34 UTC
Um...

      def __init__(self, host=None, user=None, password="",
                   database=None, port=0, unix_socket=None,
                   charset='', sql_mode=None,
                   read_default_file=None, conv=None, use_unicode=None,
                   client_flag=0, cursorclass=Cursor, init_command=None,
                   connect_timeout=10, ssl=None, read_default_group=None,
                   compress=None, named_pipe=None, no_delay=None,
                   autocommit=False, db=None, passwd=None, local_infile=False,
                   max_allowed_packet=16*1024*1024, defer_connect=False,
                   auth_plugin_map={}, read_timeout=None, write_timeout=None,
                   bind_address=None):
                   ^^^^^^^^^^^^^^^^^

Damien, why was placing the bind_address in a file and using read_default_file used instead of just using bind_address=...?

The location of the file (/etc/my.cnf.d) makes this pretty ugly to work around in selinux policies since we'd need to allow every OpenStack service to read all MySQL configuration files or change this file to something that can be read by any context.

Comment 8 Lon Hohberger 2017-03-28 21:21:50 UTC
(That's the __init__ function from connections.py in PyMySQL)

Comment 9 Damien Ciabrini 2017-03-29 07:11:39 UTC
Lon,
 
Up to Ocata we used to set the bind_address argument directly [1], by appending the according option to the DBURI of all OpenStackService, e.g.:

   connection=mysql+pymysql://nova_api:R4TNM9XvM3Ptg7GcK3dnnHR4W@10.0.0.15/nova_api?bind_address=1.2.3.4

Unfortunately, this breaks Nova's cells v2 design, which stores this DBURI into the database itself and use it to create connection to cells database. By doing so, only the controller which holds the bind_address is able to connect to cells database (1 controllers among the 3), so this breaks HA and we needed to revert that setting [2].

The only other means we have to pass that setting to PyMySQL is to make it read it from file. We couldn't use the regular /etc/my.cnf.d/* ones due to limitation 
in PyMysql parsing [3], so we needed to come up with this dedicated tripleo.cnf [4]

Knowing that this file does not contain any secret (just client connection configuration), I think it's safe to make it unconfined for SELinux.

[1] https://review.openstack.org/#/c/414629/
[2] https://review.openstack.org/#/c/430183/
[3] https://github.com/PyMySQL/PyMySQL/issues/548
[4] https://review.openstack.org/436192

Comment 10 Damien Ciabrini 2017-03-29 12:24:10 UTC
Also, as noted in comment #6, /etc/my.cnf.d/client.cnf already has unconfined_u property; we just went for a dedicated tripleo.cnf instead of client.cnf due to PyMySQL's parser limitation.

So having the same selinux context for tripleo.cnf and client.cnf seems to be the way to go.

As for the current impact: with those SELinux denials, the bind_address setting is not read by PyMySQL, but a socket is still created for the DB connection. I.e., it shouldn't block ongoing tests, it's only breaking the original VIP failover bug [1]

[1] https://launchpad.net/tripleo/+bug/1643487

Comment 11 Lon Hohberger 2017-03-29 12:58:41 UTC
The problem is the label and location:

   mysql_etc_t

We can do:

  mysql_read_config(nova_t)
  mysql_read_config(nova_api_t)
  mysql_read_config(neutron_t)
  ...
  mysql_read_config(glance_t)
  ...

Then, they can read:

  /etc/my.cnf
  /etc/my.cnf.d/*

It actually looks okay to do this, it's just messy.  However, it appears other system daemons do this, including Neutron:

  quantum.te:	mysql_read_config(neutron_t)

Comment 12 Lon Hohberger 2017-03-29 13:00:59 UTC
Presently:

[lhh@localhost selinux-policy-contrib]$ grep mysql_read_config * | grep -v ^mysql
apache.te:	mysql_read_config(httpd_t)
apache.te:	mysql_read_config(httpd_php_t)
apache.te:	mysql_read_config(httpd_suexec_t)
apache.te:	mysql_read_config(httpd_sys_script_t)
cron.te:	mysql_read_config(system_cronjob_t)
dovecot.te:	mysql_read_config(dovecot_auth_t)
dspam.te:	mysql_read_config(dspam_t)
logrotate.te:	mysql_read_config(logrotate_t)
munin.te:	mysql_read_config(munin_t)
munin.te:	mysql_read_config(services_munin_plugin_t)
mythtv.te:	mysql_read_config(mythtv_script_t)
nagios.te:      mysql_read_config(nagios_services_plugin_t)
pdns.te:	mysql_read_config(pdns_t)
quantum.te:	mysql_read_config(neutron_t)
radius.te:	mysql_read_config(radiusd_t)

Comment 13 Lon Hohberger 2017-03-30 13:36:31 UTC
Note: not posting upstream since the log was captured in enforcing mode instead of permissive. There might be other issues.

Comment 14 Lon Hohberger 2017-03-30 13:37:07 UTC
Created attachment 1267537 [details]
Patch for observed AVCs.

Comment 15 Lon Hohberger 2017-03-30 13:38:23 UTC
If this doesn't resolve it, we'll need an upgrade run in permissive to see the new AVCs that pop up. Not all services will require access to mysql_etc_t since many are run via Apache, which already has access to read mysql_etc_t.

Comment 17 errata-xmlrpc 2017-05-17 20:06:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.