1420134 – Selinux is preventing neutron-openvswitch-agent from starting properly

Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.

Bug 1420134 - Selinux is preventing neutron-openvswitch-agent from starting properly

Summary: Selinux is preventing neutron-openvswitch-agent from starting properly

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-selinux
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	z3
Target Release:	10.0 (Newton)
Assignee:	Lon Hohberger
QA Contact:	David Hill
Docs Contact:
URL:
Whiteboard:
Depends On:	1419418
Blocks:
TreeView+	depends on / blocked

Reported:	2017-02-07 23:33 UTC by David Hill
Modified:	2017-07-11 06:59 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openstack-selinux-0.8.5-1.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1419418
Environment:
Last Closed:	2017-06-28 15:27:21 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1649124	0	None	None	None	2017-02-07 23:33:42 UTC
Red Hat Product Errata	RHBA-2017:1587	0	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 10 Bug Fix and Enhancement Advisory	2017-06-28 19:11:42 UTC

Description David Hill 2017-02-07 23:33:42 UTC

+++ This bug was initially created as a clone of Bug #1419418 +++

Description of problem:
Selinux is preventing neutron-openvswitch-agent from starting properly .  When the issue occurs, ovs-vswitchd is using 100% CPU and unless "semodule -R -D" is ran, no selinux messages appears but ovs-vswitchd is logging lots of permission denied entries.  For what it worths, here is a quick fix:


policy_module(fix,0.0.1)
require {
 type neutron_t;
 type nsfs_t;
 type openvswitch_t;
 class netlink_generic_socket { connect create getattr getopt setopt read write };
 class file { open read };
}
allow neutron_t nsfs_t:file { open read };
allow openvswitch_t self:netlink_generic_socket { connect create getattr getopt setopt read write};
neutron_systemctl(openvswitch_t)
hostname_exec(openvswitch_t)

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Install rdo-newton/ocata/trunk repositories on RHEL 7.3 
2. Run "openstack undercloud install"
3.

Actual results:
Everything succeeds BUT neutron-openvswitch-agent dies on first startup because ovs-vsctl manager creation failed with -14.  Lots of logs in ovs-vswitchd with permission denied and overcloud creation will fail.

Expected results:
Selinux shouldn't be a problem with newton/ocata

Additional info:
This is easily reprocible

--- Additional comment from David Hill on 2017-02-06 00:58:53 EST ---

--- a/os-neutron.te
+++ b/os-neutron.te
@@ -13,6 +13,7 @@ gen_require(`
        type proc_t;
        type radvd_exec_t;
        type modules_object_t;
+  type nsfs_t;
        type ipsec_key_file_t;
        type keepalived_t;
        type logrotate_t;
@@ -26,6 +27,7 @@ gen_require(`
        class unix_stream_socket connectto;
        class dir search;
        class netlink_selinux_socket create;
+  class netlink_generic_socket { connect create getattr getopt setopt read write };
 ')
 
 # Bugzilla 1357961
@@ -34,6 +36,12 @@ corenet_tcp_bind_openflow_port(neutron_t)
 # Bugzilla 1180679
 allow neutron_t keepalived_t:process signal;
 
+# Bugzilla 1419418
+allow neutron_t nsfs_t:file { open read };
+allow openvswitch_t self:netlink_generic_socket { connect create getattr getopt setopt read write};
+neutron_systemctl(openvswitch_t)
+hostname_exec(openvswitch_t)
+
 # Bugzilla 1168526 & 1176830
 allow neutron_t radvd_exec_t:file { read open execute execute_no_trans };
 fs_getattr_all_fs(neutron_t)

--- Additional comment from Ihar Hrachyshka on 2017-02-06 14:28:26 EST ---

We already have a bunch of rules in https://github.com/redhat-openstack/openstack-selinux/blob/el7/os-ovs.te that seem related. So why don't they work for you? Do you have them included into your openstack-selinux package? Is your vswitchd started before the rules applied? Lots of questions to answer before we open another hole in our security wall.

Please attach service logs for neutron, as well as syslog, as well as audit.log, and config files for neutron services.

It's also not clear why the patch that you proposed to Neutron that masked quotes helped your case. I can't find how this is relevant to the selinux rule you suggested.

I can't believe native interface is broken in RDO, it's default option; as per Brent Eagles, it's not overridden by either puppet or heat or tripleo, and we have tempest jobs in RDO CI pipeline that would catch an error like not being able to add the manager (that would make L2 agent completely disfunctional).

So, in essence, please give more details about your setup, and logs, and maybe rationale on the rules you suggest (how have you come up to them?)

--- Additional comment from David Hill on 2017-02-07 17:10 EST ---



--- Additional comment from David Hill on 2017-02-07 17:11:13 EST ---

Name        : openstack-selinux
Version     : 0.7.13
Release     : 2.el7
Architecture: noarch
Install Date: Tue 07 Feb 2017 11:27:53 AM EST
Group       : System Environment/Base
Size        : 155383
License     : GPLv2
Signature   : (none)
Source RPM  : openstack-selinux-0.7.13-2.el7.src.rpm
Build Date  : Fri 09 Dec 2016 09:06:24 AM EST
Build Host  : c1bj.rdu2.centos.org
Relocations : (not relocatable)
Packager    : CBS <cbs@centos.org>
Vendor      : CentOS
URL         : https://github.com/redhat-openstack/openstack-selinux
Summary     : SELinux Policies for OpenStack
Description :
SELinux policy modules for use with OpenStack

--- Additional comment from David Hill on 2017-02-07 17:14:26 EST ---

[root@undercloud-0-newton audit]# semodule -l | grep ovs
os-ovs	0.1

--- Additional comment from David Hill on 2017-02-07 18:29:39 EST ---

I seem to be able to reproduce this with RHEL 7.3 and RHOSP 10 at this point.  I'm using the latest RHEL 7.3 KVM image and spawn RHOSP 10 / RDO-Newton/Ocata/Etc from a systemd init script.  Maybe our RHEL 7.3 KVM image has issues?

--- Additional comment from David Hill on 2017-02-07 18:32:42 EST ---

Name        : openstack-selinux
Arch        : noarch
Version     : 0.7.13
Release     : 3.el7ost
Size        : 152 k
Repo        : installed
From repo   : rhelosp-10.0-puddle
Summary     : SELinux Policies for OpenStack
URL         : https://github.com/redhat-openstack/openstack-selinux
License     : GPLv2
Description : SELinux policy modules for use with OpenStack


# cat audit.log | grep denied | audit2allow -R
require {
	type openvswitch_t;
	class netlink_generic_socket getopt;
}

#============= openvswitch_t ==============
allow openvswitch_t self:netlink_generic_socket getopt;



One of the main symptoms I can see here is when ovs-vswitched is using 100% CPU  and ovs-vswitchd logs lots of the following:
2017-02-07T23:25:14.921Z|02933|netlink_socket|ERR|transaction error (Permission denied)


This issue is really openstack-selinux not allowing the proper syscall and needs to be whitelisted.

Comment 1 Mike Burns 2017-02-08 11:11:57 UTC

Please provide the full audit.log while running in permissive mode.  That will help determine the best path forward.  audit2allow doesn't always provide the best option.

Comment 2 David Hill 2017-02-08 17:09:40 UTC

The probleme here is that openstack undercloud install switches it back to enforcing along the way.  In the RHOSP 10.0 I only had that line that I can simply copy/paste it here (but it was repeaded 100000 times).

Comment 3 David Hill 2017-02-10 00:16:02 UTC

type=AVC msg=audit(1486685192.236:18260264): avc:  denied  { search } for  pid=7040 comm="ovs-vsctl" name="7032" dev="proc" ino=499867 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1486685192.236:18260264): avc:  denied  { read } for  pid=7040 comm="ovs-vsctl" name="cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685192.236:18260264): avc:  denied  { open } for  pid=7040 comm="ovs-vsctl" path="/proc/7032/cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685192.236:18260265): avc:  denied  { getattr } for  pid=7040 comm="ovs-vsctl" path="/proc/7032/cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685192.341:18260268): avc:  denied  { create } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260269): avc:  denied  { setopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260270): avc:  denied  { getopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260271): avc:  denied  { connect } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260272): avc:  denied  { getattr } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685208.812:18260753): avc:  denied  { write } for  pid=7605 comm="iptables-save" path="/etc/sysconfig/iptables" dev="vda1" ino=7458118 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.763:18261281): avc:  denied  { read } for  pid=7989 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.763:18261281): avc:  denied  { open } for  pid=7989 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.914:18261285): avc:  denied  { create } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261286): avc:  denied  { setopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261287): avc:  denied  { getopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261288): avc:  denied  { connect } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261289): avc:  denied  { getattr } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.983:18261295): avc:  denied  { read } for  pid=8003 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.983:18261295): avc:  denied  { open } for  pid=8003 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685224.408:18261328): avc:  denied  { read } for  pid=8059 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685224.408:18261328): avc:  denied  { open } for  pid=8059 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685225.005:18261366): avc:  denied  { read } for  pid=8075 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685225.005:18261366): avc:  denied  { open } for  pid=8075 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1

Comment 4 David Hill 2017-02-14 15:20:04 UTC

Quick workaround for me is to put selinux in permissive in RedHat.yaml like Centos.yaml:

sed -i 's/enforcing/permissive/' /usr/share/instack-undercloud/puppet-stack-config/os-apply-config/etc/puppet/hieradata/RedHat.yaml

This is why CI is not hitting this issue if anyone asks and there're probably selinux audit logs for that.

Comment 5 Lon Hohberger 2017-02-20 23:24:07 UTC

https://github.com/redhat-openstack/openstack-selinux/commit/27bd924074aee23c7cb9540e182f0cd0da6da3fd

Comment 8 nlevinki 2017-06-27 14:45:32 UTC

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP10/

Comment 10 errata-xmlrpc 2017-06-28 15:27:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1587

Note You need to log in before you can comment on or make changes to this bug.