+++ This bug was initially created as a clone of Bug #1419418 +++ Description of problem: Selinux is preventing neutron-openvswitch-agent from starting properly . When the issue occurs, ovs-vswitchd is using 100% CPU and unless "semodule -R -D" is ran, no selinux messages appears but ovs-vswitchd is logging lots of permission denied entries. For what it worths, here is a quick fix: policy_module(fix,0.0.1) require { type neutron_t; type nsfs_t; type openvswitch_t; class netlink_generic_socket { connect create getattr getopt setopt read write }; class file { open read }; } allow neutron_t nsfs_t:file { open read }; allow openvswitch_t self:netlink_generic_socket { connect create getattr getopt setopt read write}; neutron_systemctl(openvswitch_t) hostname_exec(openvswitch_t) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Install rdo-newton/ocata/trunk repositories on RHEL 7.3 2. Run "openstack undercloud install" 3. Actual results: Everything succeeds BUT neutron-openvswitch-agent dies on first startup because ovs-vsctl manager creation failed with -14. Lots of logs in ovs-vswitchd with permission denied and overcloud creation will fail. Expected results: Selinux shouldn't be a problem with newton/ocata Additional info: This is easily reprocible --- Additional comment from David Hill on 2017-02-06 00:58:53 EST --- --- a/os-neutron.te +++ b/os-neutron.te @@ -13,6 +13,7 @@ gen_require(` type proc_t; type radvd_exec_t; type modules_object_t; + type nsfs_t; type ipsec_key_file_t; type keepalived_t; type logrotate_t; @@ -26,6 +27,7 @@ gen_require(` class unix_stream_socket connectto; class dir search; class netlink_selinux_socket create; + class netlink_generic_socket { connect create getattr getopt setopt read write }; ') # Bugzilla 1357961 @@ -34,6 +36,12 @@ corenet_tcp_bind_openflow_port(neutron_t) # Bugzilla 1180679 allow neutron_t keepalived_t:process signal; +# Bugzilla 1419418 +allow neutron_t nsfs_t:file { open read }; +allow openvswitch_t self:netlink_generic_socket { connect create getattr getopt setopt read write}; +neutron_systemctl(openvswitch_t) +hostname_exec(openvswitch_t) + # Bugzilla 1168526 & 1176830 allow neutron_t radvd_exec_t:file { read open execute execute_no_trans }; fs_getattr_all_fs(neutron_t) --- Additional comment from Ihar Hrachyshka on 2017-02-06 14:28:26 EST --- We already have a bunch of rules in https://github.com/redhat-openstack/openstack-selinux/blob/el7/os-ovs.te that seem related. So why don't they work for you? Do you have them included into your openstack-selinux package? Is your vswitchd started before the rules applied? Lots of questions to answer before we open another hole in our security wall. Please attach service logs for neutron, as well as syslog, as well as audit.log, and config files for neutron services. It's also not clear why the patch that you proposed to Neutron that masked quotes helped your case. I can't find how this is relevant to the selinux rule you suggested. I can't believe native interface is broken in RDO, it's default option; as per Brent Eagles, it's not overridden by either puppet or heat or tripleo, and we have tempest jobs in RDO CI pipeline that would catch an error like not being able to add the manager (that would make L2 agent completely disfunctional). So, in essence, please give more details about your setup, and logs, and maybe rationale on the rules you suggest (how have you come up to them?) --- Additional comment from David Hill on 2017-02-07 17:10 EST --- --- Additional comment from David Hill on 2017-02-07 17:11:13 EST --- Name : openstack-selinux Version : 0.7.13 Release : 2.el7 Architecture: noarch Install Date: Tue 07 Feb 2017 11:27:53 AM EST Group : System Environment/Base Size : 155383 License : GPLv2 Signature : (none) Source RPM : openstack-selinux-0.7.13-2.el7.src.rpm Build Date : Fri 09 Dec 2016 09:06:24 AM EST Build Host : c1bj.rdu2.centos.org Relocations : (not relocatable) Packager : CBS <cbs> Vendor : CentOS URL : https://github.com/redhat-openstack/openstack-selinux Summary : SELinux Policies for OpenStack Description : SELinux policy modules for use with OpenStack --- Additional comment from David Hill on 2017-02-07 17:14:26 EST --- [root@undercloud-0-newton audit]# semodule -l | grep ovs os-ovs 0.1 --- Additional comment from David Hill on 2017-02-07 18:29:39 EST --- I seem to be able to reproduce this with RHEL 7.3 and RHOSP 10 at this point. I'm using the latest RHEL 7.3 KVM image and spawn RHOSP 10 / RDO-Newton/Ocata/Etc from a systemd init script. Maybe our RHEL 7.3 KVM image has issues? --- Additional comment from David Hill on 2017-02-07 18:32:42 EST --- Name : openstack-selinux Arch : noarch Version : 0.7.13 Release : 3.el7ost Size : 152 k Repo : installed From repo : rhelosp-10.0-puddle Summary : SELinux Policies for OpenStack URL : https://github.com/redhat-openstack/openstack-selinux License : GPLv2 Description : SELinux policy modules for use with OpenStack # cat audit.log | grep denied | audit2allow -R require { type openvswitch_t; class netlink_generic_socket getopt; } #============= openvswitch_t ============== allow openvswitch_t self:netlink_generic_socket getopt; One of the main symptoms I can see here is when ovs-vswitched is using 100% CPU and ovs-vswitchd logs lots of the following: 2017-02-07T23:25:14.921Z|02933|netlink_socket|ERR|transaction error (Permission denied) This issue is really openstack-selinux not allowing the proper syscall and needs to be whitelisted.
Please provide the full audit.log while running in permissive mode. That will help determine the best path forward. audit2allow doesn't always provide the best option.
The probleme here is that openstack undercloud install switches it back to enforcing along the way. In the RHOSP 10.0 I only had that line that I can simply copy/paste it here (but it was repeaded 100000 times).
type=AVC msg=audit(1486685192.236:18260264): avc: denied { search } for pid=7040 comm="ovs-vsctl" name="7032" dev="proc" ino=499867 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=dir permissive=1 type=AVC msg=audit(1486685192.236:18260264): avc: denied { read } for pid=7040 comm="ovs-vsctl" name="cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685192.236:18260264): avc: denied { open } for pid=7040 comm="ovs-vsctl" path="/proc/7032/cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685192.236:18260265): avc: denied { getattr } for pid=7040 comm="ovs-vsctl" path="/proc/7032/cmdline" dev="proc" ino=497396 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:neutron_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685192.341:18260268): avc: denied { create } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685192.341:18260269): avc: denied { setopt } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685192.341:18260270): avc: denied { getopt } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685192.341:18260271): avc: denied { connect } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685192.341:18260272): avc: denied { getattr } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685208.812:18260753): avc: denied { write } for pid=7605 comm="iptables-save" path="/etc/sysconfig/iptables" dev="vda1" ino=7458118 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685222.763:18261281): avc: denied { read } for pid=7989 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685222.763:18261281): avc: denied { open } for pid=7989 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685222.914:18261285): avc: denied { create } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685222.915:18261286): avc: denied { setopt } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685222.915:18261287): avc: denied { getopt } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685222.915:18261288): avc: denied { connect } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685222.915:18261289): avc: denied { getattr } for pid=29183 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=1 type=AVC msg=audit(1486685222.983:18261295): avc: denied { read } for pid=8003 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685222.983:18261295): avc: denied { open } for pid=8003 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685224.408:18261328): avc: denied { read } for pid=8059 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685224.408:18261328): avc: denied { open } for pid=8059 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685225.005:18261366): avc: denied { read } for pid=8075 comm="ip" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1 type=AVC msg=audit(1486685225.005:18261366): avc: denied { open } for pid=8075 comm="ip" path="net:[4026531957]" dev="nsfs" ino=4026531957 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:nsfs_t:s0 tclass=file permissive=1
Quick workaround for me is to put selinux in permissive in RedHat.yaml like Centos.yaml: sed -i 's/enforcing/permissive/' /usr/share/instack-undercloud/puppet-stack-config/os-apply-config/etc/puppet/hieradata/RedHat.yaml This is why CI is not hitting this issue if anyone asks and there're probably selinux audit logs for that.
https://github.com/redhat-openstack/openstack-selinux/commit/27bd924074aee23c7cb9540e182f0cd0da6da3fd
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP10/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1587