Description of problem: We are seeing avc denials related to "net_broadcast" in OSP14 deployments on OC nodes. They do not seem to be harmful for deployment itself and full Tempest testing, but we need to check whether there could be some action taken to get rid of them. Example: type=AVC msg=audit(...): avc: denied { net_broadcast } for pid=... comm=\"ovs-vswitchd\" capability=11 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability permissive=0 Version-Release number of selected component (if applicable): OSP14, RHEL 7.5, 7.6 Steps to Reproduce: 1. Deploy OSP14 using InfraRed 2. Check /var/log/audit/audit.log for avc denials on OC nodes (undercloud doesn't seem to be affected) Actual results: There is one such denial in the log for each OC node, total amount per deployment is based on amount of OC nodes deployed. Expected results: There are no avc denials on any node. Additional info: # example from affected controller openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch openstack-selinux-0.8.15-0.20181003125347.4a047a0.el7ost.noarch
Note this might be related/remains of https://bugzilla.redhat.com/show_bug.cgi?id=1608508 . Also, the missing rules seems to be: $ cat /var/log/audit/audit.log | audit2allow #============= openvswitch_t ============== allow openvswitch_t self:capability net_broadcast;
The fix specified in c#2 is actually applied as a part of selinux policy version 1.0-4 (and so should be included). http://pkgs.devel.redhat.com/cgit/rpms/openvswitch-selinux-extra-policy/commit/?h=fast-datapath-rhel-7&id=68cffbfdeed935c9bc054b92d65d4853471f6f99 Are you sure the openvswitch selinux rpm has been installed in the host?
(In reply to Aaron Conole from comment #3) > The fix specified in c#2 is actually applied as a part of selinux policy > version 1.0-4 (and so should be included). > > http://pkgs.devel.redhat.com/cgit/rpms/openvswitch-selinux-extra-policy/ > commit/?h=fast-datapath-rhel-7&id=68cffbfdeed935c9bc054b92d65d4853471f6f99 > > Are you sure the openvswitch selinux rpm has been installed in the host? Looking at affected compute node, we have openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch on both host and inside neutron_ovs_agent container.
Is it possible that you are starting OvS before installing the selinux policy package? That's about the only think I can think is happening?
(In reply to Aaron Conole from comment #5) > Is it possible that you are starting OvS before installing the selinux > policy package? That's about the only think I can think is happening? I don't think that is the case. These nodes are part of overcloud deployment, I can access them when they become available during early stages of OC deployment (network becomes available) - the AVC deny record is already present in audit.log at that time without any mention that openvswitch-selinux-extra-policy rpm being (not)installed(yet) after first boot. Moreover, if I extract contents of rhosp-director-images directly on undercloud manually, overcloud-full.qcow2 image contains /usr/share/selinux/packages/openvswitch-custom.pp file pre-installed: libguestfs | overcloud-full.qcow2 $ rpm -ql openvswitch-selinux-extra-policy /usr/share/selinux/packages/openvswitch-custom.pp so I assume openvswitch-selinux-extra-policy package is already part of OC images, not installed as part of the deployment of OC nodes.
I just confirmed with a fresh rhel7.6 VM: http://download-node-02.eng.bos.redhat.com/brewroot/packages/rhel-guest-image/7.6/210/images/rhel-guest-image-7.6-210.x86_64.qcow2 I installed on that image: yum install http://download-node-02.eng.bos.redhat.com/brewroot/packages/openvswitch2.10/2.10.0/4.el7fdp/x86_64/openvswitch2.10-2.10.0-4.el7fdp.x86_64.rpm yum install http://download-node-02.eng.bos.redhat.com/brewroot/packages/openvswitch2.10/2.10.0/4.el7fdp/x86_64/openvswitch2.10-2.10.0-4.el7fdp.x86_64.rpm http://download-node-02.eng.bos.redhat.com/brewroot/packages/unbound/1.6.6/1.el7/x86_64/unbound-libs-1.6.6-1.el7.x86_64.rpm Then: [root@localhost ~]# getenforce Enforcing [root@localhost ~]# systemctl start openvswitch [root@localhost ~]# ps aux | grep ovs openvsw+ 3241 0.0 0.3 156980 3996 ? S<s 16:10 0:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach openvsw+ 3334 0.0 2.2 159652 22580 ? S<Ls 16:10 0:00 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach root 3363 0.0 0.0 112708 944 pts/0 S+ 16:12 0:00 grep --color=auto ovs [root@localhost ~]# grep AVC /var/log/audit/audit.log [root@localhost ~]# grep net_broad /var/log/audit/audit.log Please give a minimum viable reproducer.
Using the same image rhel-guest-image-7.6-210.x86_64.qcow2 as base image for whole OSP14 virthost deployment, my output is following (compute node, note we have newer openvswitch2.10-2.10.0-21.el7fdn): [root@compute-0 audit]# getenforce Enforcing [root@compute-0 audit]# rpm -qa | grep openv openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch rhosp-openvswitch-2.10-0.1.el7ost.noarch openvswitch2.10-2.10.0-21.el7fdn.x86_64 [root@compute-0 audit]# rpm -qa | grep unbound unbound-libs-1.6.6-1.el7.x86_64 [root@compute-0 audit]# >audit.log [root@compute-0 audit]# date Thu Nov 1 09:10:20 UTC 2018 [root@compute-0 audit]# systemctl restart openvswitch [root@compute-0 audit]# cat audit.log | grep denied | ausearch -i ---- type=AVC msg=audit(11/01/2018 09:10:36.848:9787) : avc: denied { net_broadcast } for pid=1045325 comm=ovs-vswitchd capability=net_broadcast scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability permissive=0 Tested on compute node, where I can see AVC denial surfaced during deployment. I will attach differential audit.log just for "systemctl restart openvswitch" operation also with output of "ps aux | grep ovs" included.
Created attachment 1499880 [details] audit.log.and.ps#comment9
There is a lot running on your system in question. If I take the bare system, install the selinux policy, and openvswitch2.10, I cannot reproduce this error. On your system, you have much more packages installed. Is it possible that you have some kind of configuration that has suppressed the selinux policy? Regardless, this isn't a bug: neither the policy, nor the package are at issue here. Requesting again: please give me the steps from start to finish to go from a base image to this issue. I have given you the steps I attempted to reproduce (minimal image, install selinux policy, and openvswitch packages) - I was unsuccessful.
I can not provide steps to reproduce this issue from base image(7.6, v210) using simple approach from comment#8, which makes this TripleO/OVS bug. This BZ is targeted against OpenStack overcloud environment and happens only inside such. I deployed reproduced OSP14 environment, access to it is described in attached trello escalation ticket: https://trello.com/c/T9byGPLl/798-cixbz1643571ospphase2osp14openvswitchavc-denied-netbroadcast-for-pid-commovs-vswitchd .
As noted by Filip in comment 8, this is not a race condition at installation, as this avc can be observed whenever restarting openvswitch ps output looks correct: # ps -eZ | grep openvswitch_t system_u:system_r:openvswitch_t:s0 426154 ? 00:00:00 ovsdb-server system_u:system_r:openvswitch_t:s0 426206 ? 00:00:01 ovs-vswitchd Issuing restorecon on ovs-vswitchd does not fix the issue Selinux related packages: # rpm -qa|grep selinux libselinux-ruby-2.5-14.1.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libselinux-utils-2.5-14.1.el7.x86_64 selinux-policy-3.13.1-229.el7_6.5.noarch selinux-policy-targeted-3.13.1-229.el7_6.5.noarch openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch libselinux-python-2.5-14.1.el7.x86_64 container-selinux-2.73-2.el7.noarch openstack-selinux-0.8.15-0.20181003125347.4a047a0.el7ost.noarch One possible idea here is that openstack-selinux changes to policy may reset the net_broadcast capability allowed by openvswitch-selinux-extra-policy?
Even if I manually install the openstack-selinux and container-selinux packages, I cannot reproduce this. This is something specific to your environment. I'm going to assign this to another component. Without any further details, the only thing I can say is some installer script or otherwise is breaking this.
reinstalling openvswitch-selinux-extra-policy appears to clear up the denied issue. I'm wondering if something is getting lost in the image building process. I tried reinstalling the other selinux policy rpms to see if it's an ordering thing but so far no luck.
semodule -i /usr/share/selinux/packages/openvswitch-custom.pp also clears up the error
confirmed - audit2why reports: type=AVC msg=audit(...): avc: denied { net_broadcast } for pid=... comm=\"ovs-vswitchd\" capability=11 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=capability permissive=0 Was caused by: Unknown - would be allowed by active policy Possible mismatch between this policy and the one under which the audit message was generated. Possible mismatch between current in-memory boolean settings vs. permanent ones. [root@localhost ~]# rpm -q openstack-selinux selinux-policy openvswitch-selinux-extra-policy openstack-selinux-0.8.15-1.el7ost.noarch selinux-policy-3.13.1-229.el7_6.5.noarch openvswitch-selinux-extra-policy-1.0-5.el7fdp.noarch
Ah, the image build process... Right. Thanks. That's the ticket. There's careful weird things in openstack-selinux to work with image builds; let me look at openvswitch-selinux-extra-policy.
This is related to bug 1609323 - but not entirely explained by it. Basically, installing modules and setting things up needs to be handled differently when building images than when installing a package on a running machine. This is because during an image build, SELinux is disabled - and access to the kernel store is not allowed. However, upon boot of the image, we want the policies to be loaded, and the on-disk file contexts to match what is expected. The SELinux macros only work if SELinux is enabled, so they cannot be used. So, if we are going to build SELinux policy packages that are to work in images, we need to do things like: %post: semodule -N -i /path/to/<module>.pp [ -s <selinux_type> ] if selinuxenabled; then load_policy fi %preun: semodule -N -r <module> [ -s <selinux_type> ] if selinuxenabled; then load_policy fi For setting file contexts, we need to (unconditionally) do this in %post after installing our policy modules: setfiles -F /etc/selinux/targeted/contexts/files/file_contexts \ /path/1 \ /path/2 \ /whatever This reads the SELinux file context store and sets file contexts whether or not selinux is enabled (contrast to 'restorecon', which must have the policy loaded into the kernel). Presently, openvswitch-selinux-extra-policy does not have any custom file contexts, so there is no need to do this right now.
Created attachment 1509551 [details] Spec file fix for openvswitch-selinux-extra-policy
Brew builds for rhel7 and rhel8 with the proposed workaround applied: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=19305595 https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=19305620 Thanks for the explanation, Lon!
Verified by CI, since puddle 2018-11-29.3 and openvswitch-selinux-extra-policy-1.0-9.el7fdp included we don't see these AVC denials on OC nodes anymore.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045