Bug 1420134

Summary: Selinux is preventing neutron-openvswitch-agent from starting properly
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-selinuxAssignee: Lon Hohberger <lhh>
Status: CLOSED ERRATA QA Contact: David Hill <dhill>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: dhill, ihrachys, lhh, mburns, mgrepl, nlevinki, oblaut, srevivo
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-selinux-0.8.5-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1419418 Environment:
Last Closed: 2017-06-28 15:27:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1419418    
Bug Blocks:    

Description David Hill 2017-02-07 23:33:42 UTC 
+++ This bug was initially created as a clone of Bug #1419418 +++

Description of problem:
Selinux is preventing neutron-openvswitch-agent from starting properly .  When the issue occurs, ovs-vswitchd is using 100% CPU and unless "semodule -R -D" is ran, no selinux messages appears but ovs-vswitchd is logging lots of permission denied entries.  For what it worths, here is a quick fix:


policy_module(fix,0.0.1)
require {
 type neutron_t;
 type nsfs_t;
 type openvswitch_t;
 class netlink_generic_socket { connect create getattr getopt setopt read write };
 class file { open read };
}
allow neutron_t nsfs_t:file { open read };
allow openvswitch_t self:netlink_generic_socket { connect create getattr getopt setopt read write};
neutron_systemctl(openvswitch_t)
hostname_exec(openvswitch_t)

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Install rdo-newton/ocata/trunk repositories on RHEL 7.3 
2. Run "openstack undercloud install"
3.

Actual results:
Everything succeeds BUT neutron-openvswitch-agent dies on first startup because ovs-vsctl manager creation failed with -14.  Lots of logs in ovs-vswitchd with permission denied and overcloud creation will fail.

Expected results:
Selinux shouldn't be a problem with newton/ocata

Additional info:
This is easily reprocible

--- Additional comment from David Hill on 2017-02-06 00:58:53 EST ---

--- a/os-neutron.te
+++ b/os-neutron.te
@@ -13,6 +13,7 @@ gen_require(`
        type proc_t;
        type radvd_exec_t;
        type modules_object_t;
+  type nsfs_t;
        type ipsec_key_file_t;
        type keepalived_t;
        type logrotate_t;
@@ -26,6 +27,7 @@ gen_require(`
        class unix_stream_socket connectto;
        class dir search;
        class netlink_selinux_socket create;
+  class netlink_generic_socket { connect create getattr getopt setopt read write };
 ')
 
 # Bugzilla 1357961
@@ -34,6 +36,12 @@ corenet_tcp_bind_openflow_port(neutron_t)
 # Bugzilla 1180679
 allow neutron_t keepalived_t:process signal;
 
+# Bugzilla 1419418
+allow neutron_t nsfs_t:file { open read };
+allow openvswitch_t self:netlink_generic_socket { connect create getattr getopt setopt read write};
+neutron_systemctl(openvswitch_t)
+hostname_exec(openvswitch_t)
+
 # Bugzilla 1168526 & 1176830
 allow neutron_t radvd_exec_t:file { read open execute execute_no_trans };
 fs_getattr_all_fs(neutron_t)

--- Additional comment from Ihar Hrachyshka on 2017-02-06 14:28:26 EST ---

We already have a bunch of rules in https://github.com/redhat-openstack/openstack-selinux/blob/el7/os-ovs.te that seem related. So why don't they work for you? Do you have them included into your openstack-selinux package? Is your vswitchd started before the rules applied? Lots of questions to answer before we open another hole in our security wall.

Please attach service logs for neutron, as well as syslog, as well as audit.log, and config files for neutron services.

It's also not clear why the patch that you proposed to Neutron that masked quotes helped your case. I can't find how this is relevant to the selinux rule you suggested.

I can't believe native interface is broken in RDO, it's default option; as per Brent Eagles, it's not overridden by either puppet or heat or tripleo, and we have tempest jobs in RDO CI pipeline that would catch an error like not being able to add the manager (that would make L2 agent completely disfunctional).

So, in essence, please give more details about your setup, and logs, and maybe rationale on the rules you suggest (how have you come up to them?)

--- Additional comment from David Hill on 2017-02-07 17:10 EST ---



--- Additional comment from David Hill on 2017-02-07 17:11:13 EST ---

Name        : openstack-selinux
Version     : 0.7.13
Release     : 2.el7
Architecture: noarch
Install Date: Tue 07 Feb 2017 11:27:53 AM EST
Group       : System Environment/Base
Size        : 155383
License     : GPLv2
Signature   : (none)
Source RPM  : openstack-selinux-0.7.13-2.el7.src.rpm
Build Date  : Fri 09 Dec 2016 09:06:24 AM EST
Build Host  : c1bj.rdu2.centos.org
Relocations : (not relocatable)
Packager    : CBS <cbs>
Vendor      : CentOS
URL         : https://github.com/redhat-openstack/openstack-selinux
Summary     : SELinux Policies for OpenStack
Description :
SELinux policy modules for use with OpenStack

--- Additional comment from David Hill on 2017-02-07 17:14:26 EST ---

[root@undercloud-0-newton audit]# semodule -l | grep ovs
os-ovs	0.1

--- Additional comment from David Hill on 2017-02-07 18:29:39 EST ---

I seem to be able to reproduce this with RHEL 7.3 and RHOSP 10 at this point.  I'm using the latest RHEL 7.3 KVM image and spawn RHOSP 10 / RDO-Newton/Ocata/Etc from a systemd init script.  Maybe our RHEL 7.3 KVM image has issues?

--- Additional comment from David Hill on 2017-02-07 18:32:42 EST ---

Name        : openstack-selinux
Arch        : noarch
Version     : 0.7.13
Release     : 3.el7ost
Size        : 152 k
Repo        : installed
From repo   : rhelosp-10.0-puddle
Summary     : SELinux Policies for OpenStack
URL         : https://github.com/redhat-openstack/openstack-selinux
License     : GPLv2
Description : SELinux policy modules for use with OpenStack


# cat audit.log | grep denied | audit2allow -R
require {
	type openvswitch_t;
	class netlink_generic_socket getopt;
}

#============= openvswitch_t ==============
allow openvswitch_t self:netlink_generic_socket getopt;



One of the main symptoms I can see here is when ovs-vswitched is using 100% CPU  and ovs-vswitchd logs lots of the following:
2017-02-07T23:25:14.921Z|02933|netlink_socket|ERR|transaction error (Permission denied)


This issue is really openstack-selinux not allowing the proper syscall and needs to be whitelisted.
Comment 1 Mike Burns 2017-02-08 11:11:57 UTC 
Please provide the full audit.log while running in permissive mode.  That will help determine the best path forward.  audit2allow doesn't always provide the best option.
Comment 2 David Hill 2017-02-08 17:09:40 UTC 
The probleme here is that openstack undercloud install switches it back to enforcing along the way.  In the RHOSP 10.0 I only had that line that I can simply copy/paste it here (but it was repeaded 100000 times).
Comment 3 David Hill 2017-02-10 00:16:02 UTC 
type=AVC msg=audit(1486685192.236:18260264): avc:  denied  { search } for  pid=7040 comm="ovs-vsctl" name="7032" dev="proc" ino=499867 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1486685192.236:18260264): avc:  denied  { read } for  pid=7040 comm="ovs-vsctl" name="cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685192.236:18260264): avc:  denied  { open } for  pid=7040 comm="ovs-vsctl" path="/proc/7032/cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685192.236:18260265): avc:  denied  { getattr } for  pid=7040 comm="ovs-vsctl" path="/proc/7032/cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685192.341:18260268): avc:  denied  { create } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260269): avc:  denied  { setopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260270): avc:  denied  { getopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260271): avc:  denied  { connect } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685192.341:18260272): avc:  denied  { getattr } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685208.812:18260753): avc:  denied  { write } for  pid=7605 comm="iptables-save" path="/etc/sysconfig/iptables" dev="vda1" ino=7458118 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.763:18261281): avc:  denied  { read } for  pid=7989 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.763:18261281): avc:  denied  { open } for  pid=7989 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.914:18261285): avc:  denied  { create } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261286): avc:  denied  { setopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261287): avc:  denied  { getopt } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261288): avc:  denied  { connect } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.915:18261289): avc:  denied  { getattr } for  pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1
type=AVC msg=audit(1486685222.983:18261295): avc:  denied  { read } for  pid=8003 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685222.983:18261295): avc:  denied  { open } for  pid=8003 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685224.408:18261328): avc:  denied  { read } for  pid=8059 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685224.408:18261328): avc:  denied  { open } for  pid=8059 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685225.005:18261366): avc:  denied  { read } for  pid=8075 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1486685225.005:18261366): avc:  denied  { open } for  pid=8075 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
Comment 4 David Hill 2017-02-14 15:20:04 UTC 
Quick workaround for me is to put selinux in permissive in RedHat.yaml like Centos.yaml:

sed -i 's/enforcing/permissive/' /usr/share/instack-undercloud/puppet-stack-config/os-apply-config/etc/puppet/hieradata/RedHat.yaml

This is why CI is not hitting this issue if anyone asks and there're probably selinux audit logs for that.
Comment 5 Lon Hohberger 2017-02-20 23:24:07 UTC 
https://github.com/redhat-openstack/openstack-selinux/commit/27bd924074aee23c7cb9540e182f0cd0da6da3fd
Comment 8 nlevinki 2017-06-27 14:45:32 UTC 
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP10/
Comment 10 errata-xmlrpc 2017-06-28 15:27:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1587