Bug 1503835 - Openvswitch crash loop when adding netdev bridge ovs 2.7.2.10 FDP and selinux enabled
Summary: Openvswitch crash loop when adding netdev bridge ovs 2.7.2.10 FDP and selinux...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy
Version: 7.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Lukas Vrabec
QA Contact: Milos Malik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-18 21:13 UTC by Christian Trautman
Modified: 2018-10-30 10:02 UTC (History)
12 users (show)

Fixed In Version: selinux-policy-3.13.1-197.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 10:01:27 UTC
Target Upstream Version:


Attachments (Terms of Use)
SOSReport (12.78 MB, application/x-xz)
2017-10-18 21:13 UTC, Christian Trautman
no flags Details
denials (5.76 MB, text/plain)
2017-11-27 20:37 UTC, Christian Trautman
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3111 None None None 2018-10-30 10:02:38 UTC

Description Christian Trautman 2017-10-18 21:13:28 UTC
Created attachment 1340384 [details]
SOSReport

Description of problem:Adding a netdev bridge to Openvswitch with selinux enabled causes Openvswitch to enter into a crash loop.

2017-10-18T21:08:17.540Z|00017|dpdk|EMER|Requested device 0000:03:00.0 cannot be used
2017-10-18T21:08:17.883Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2017-10-18T21:08:17.887Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2017-10-18T21:08:17.887Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2017-10-18T21:08:17.887Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2017-10-18T21:08:17.887Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2017-10-18T21:08:17.888Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2017-10-18T21:08:17.889Z|00007|dpdk|INFO|DPDK Enabled - initializing...
2017-10-18T21:08:17.889Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2017-10-18T21:08:17.889Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 1024,1024 -c 0x00000001
2017-10-18T21:08:17.890Z|00010|dpdk|INFO|EAL: Detected 48 lcore(s)
2017-10-18T21:08:17.913Z|00011|dpdk|INFO|EAL: Probing VFIO support...
2017-10-18T21:08:17.913Z|00012|dpdk|ERR|EAL:   cannot open VFIO container, error 13 (Permission denied)
2017-10-18T21:08:17.913Z|00013|dpdk|INFO|EAL: VFIO support could not be initialized
2017-10-18T21:08:25.543Z|00014|dpdk|INFO|EAL: PCI device 0000:03:00.0 on NUMA socket 0
2017-10-18T21:08:25.543Z|00015|dpdk|INFO|EAL:   probe driver: 8086:10fb net_ixgbe
2017-10-18T21:08:25.544Z|00016|dpdk|EMER|EAL: Error - exiting with code: 1
  Cause: 



Version-Release number of selected component (if applicable):
RHEL 7.4 3.10.0-693.5.2.el7.x86_64
openvswitch openvswitch-2.7.2-10.git20170914.el7fdp.x86_64.rpm

How reproducible:always


Steps to Reproduce:
1. Install openvswitch
2. Start openvswitch service
3. Bind a card using driverctl to vfio-pci (ixgbe card is fine)
4. Add a netdev bridge



Actual results: Openvswitch enters reboot cycle


Expected results: Openvswitch does not restart


Additional info:
SOS report attached

Comment 2 Aaron Conole 2017-10-19 13:35:51 UTC
From audit.log:

type=AVC msg=audit(1508361089.913:391): avc:  denied  { read write } for  pid=4611 comm="ovs-vswitchd" name="vfio" dev="devtmpfs" ino=18625 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:vfio_device_t:s0 tclass=chr_file

This should be fixed whenever bz 1482682 is addressed (I think).  A good test would be to take the selinux policy from upstream and test with it.

Comment 3 Christian Trautman 2017-10-19 15:07:26 UTC
I will attempt to use the upstream policy and try it.

Comment 4 Jean-Tsung Hsiao 2017-11-15 01:07:42 UTC
(In reply to Aaron Conole from comment #2)
> From audit.log:
> 
> type=AVC msg=audit(1508361089.913:391): avc:  denied  { read write } for 
> pid=4611 comm="ovs-vswitchd" name="vfio" dev="devtmpfs" ino=18625
> scontext=system_u:system_r:openvswitch_t:s0
> tcontext=system_u:object_r:vfio_device_t:s0 tclass=chr_file
> 
> This should be fixed whenever bz 1482682 is addressed (I think).  A good
> test would be to take the selinux policy from upstream and test with it.

Hi Aaron,

Bz 1482682 is for OVS-2.8. So, do you expect the new selinux policy can fix this issue.

Thanks!

Jean

Comment 5 Aaron Conole 2017-11-20 16:42:44 UTC
The policy should still work even for 2.7.  That said, I'm concerned.  Do you see this issue with OvS 2.6, also?  I would expect yes.

Comment 6 Christian Trautman 2017-11-20 19:04:03 UTC
Yes but the issue is different with 2.6. I tried version openvswitch-2.6.1-16.git20161206.el7ost.x86_64 and it just doesn't start at all with no bridges even created. 

It just enters a crash loop from failure to access the conf.db file.

Nov 20 13:58:38 localhost systemd: Starting Open vSwitch Database Unit...
Nov 20 13:58:38 localhost ovs-ctl: Starting ovsdb-server ovsdb-server: I/O error: /etc/openvswitch/conf.db: failed to lock lockfile (Resource temporarily unavailable)
Nov 20 13:58:38 localhost ovs-ctl: [FAILED]
Nov 20 13:58:38 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.14.0
Nov 20 13:58:38 localhost ovs-vsctl: ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Nov 20 13:58:38 localhost ovs-ctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Nov 20 13:58:38 localhost systemd: ovsdb-server.service: control process exited, code=exited status=1
Nov 20 13:58:38 localhost systemd: Failed to start Open vSwitch Database Unit.
Nov 20 13:58:38 localhost systemd: Unit ovsdb-server.service entered failed state.
Nov 20 13:58:38 localhost systemd: ovsdb-server.service failed.
Nov 20 13:58:38 localhost systemd: ovsdb-server.service holdoff time over, scheduling restart.

I will try your instructions to enable the policy and get back to you with that info.

Comment 7 Christian Trautman 2017-11-22 15:18:42 UTC
Hi Aaron,

I applied the upstream policy as per your instructions.

rpm -qa

openvswitch-2.7.2-10.git20170914.el7fdp.x86_64
openvswitch-selinux-policy-2.8.90-1.el7.noarch

I then tried to bind two devices to vfio-pci and start up openvswitch. It is still in a crash loop.

Comment 8 Jean-Tsung Hsiao 2017-11-23 04:05:09 UTC
(In reply to Christian Trautman from comment #7)
> Hi Aaron,
> 
> I applied the upstream policy as per your instructions.
> 
> rpm -qa
> 
> openvswitch-2.7.2-10.git20170914.el7fdp.x86_64
> openvswitch-selinux-policy-2.8.90-1.el7.noarch
> 
> I then tried to bind two devices to vfio-pci and start up openvswitch. It is
> still in a crash loop.

Hi Christ,

The openvswitch-selinux-policy package makes the difference. Below I am using a newer version, and the issue is gone.

Thanks!

Jean

[root@netqe5 ovs-2.7.2-10-testing]# rpm -q openvswitch-selinux-policy
openvswitch-selinux-policy-2.8.90-2.fc25.noarch

[root@netqe5 ovs-2.7.2-10-testing]# getenforce
Enforcing
[root@netqe5 ovs-2.7.2-10-testing]# vs
290c3841-84ea-4aa3-b259-ea43a7a5b344
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "dpdk-10"
            Interface "dpdk-10"
                type: dpdk
                options: {dpdk-devargs="0000:81:00.0", n_rxq="2"}
        Port "dpdk-11"
            Interface "dpdk-11"
                type: dpdk
                options: {dpdk-devargs="0000:81:00.1", n_rxq="2"}
    ovs_version: "2.7.2"
[root@netqe5 ovs-2.7.2-10-testing]# !ps
ps -elf | grep ovs-vs
5 S root     134931      1 99  70 -10 - 1924168 poll_s 22:54 ?      00:12:17 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
0 S root     135065 120397  0  80   0 - 28165 pipe_w 22:58 pts/0    00:00:00 grep --color=auto ovs-vs
[root@netqe5 ovs-2.7.2-10-testing]# ps -elf | grep ovs-vs
5 S root     134931      1 99  70 -10 - 1924168 poll_s 22:54 ?      00:13:12 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
0 S root     135067 120397  0  80   0 - 28165 pipe_w 22:58 pts/0    00:00:00 grep --color=auto ovs-vs

Comment 9 Milos Malik 2017-11-27 06:59:04 UTC
Could you collect all SELinux denials from the machine and attach them here?

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today

Comment 10 Milos Malik 2017-11-27 07:08:26 UTC
Based on the attached sosreport, following policy rules are needed:

allow openvswitch_t tun_tap_device_t:chr_file { read write };
allow openvswitch_t vfio_device_t:chr_file { read write };
allow virtlogd_t svirt_image_t:file unlink;
allow virtlogd_t virt_tmp_t:file unlink;

Here are the SELinux denials in raw form:
----
type=PROCTITLE msg=audit(10/18/2017 16:26:26.685:219) : proctitle=/usr/sbin/virtlogd 
type=SYSCALL msg=audit(10/18/2017 16:26:26.685:219) : arch=x86_64 syscall=unlink success=yes exit=0 a0=0x7f57dc000ce0 a1=0x7f57dc000934 a2=0x0 a3=0x2 items=0 ppid=1 pid=26050 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(10/18/2017 16:26:26.685:219) : avc:  denied  { unlink } for  pid=26050 comm=virtlogd name=master.console dev="dm-0" ino=735292 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:virt_tmp_t:s0 tclass=file
----
type=PROCTITLE msg=audit(10/18/2017 16:31:03.394:252) : proctitle=/usr/sbin/virtlogd 
type=SYSCALL msg=audit(10/18/2017 16:31:03.394:252) : arch=x86_64 syscall=unlink success=yes exit=0 a0=0x
7f57dc000ca0 a1=0x7f57dc0008d4 a2=0x0 a3=0x2 items=0 ppid=1 pid=26050 auid=unset uid=root gid=root euid=r
oot suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=virtlogd exe=/usr/sbin/
virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(10/18/2017 16:31:03.394:252) : avc:  denied  { unlink } for  pid=26050 comm=virtlogd name=master.console dev="dm-0" ino=902657 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c636,c936 tclass=file 
----
type=PROCTITLE msg=audit(10/18/2017 23:04:18.563:142) : proctitle=ovs-vswitchd unix:/var/run/openvswitch/
db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log 
type=SYSCALL msg=audit(10/18/2017 23:04:18.563:142) : arch=x86_64 syscall=open success=no exit=EACCES(Per
mission denied) a0=0x55fe00bd8d00 a1=O_RDWR a2=0x0 a3=0x3 items=0 ppid=1 pid=2624 auid=unset uid=root gid
=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ovs-vswitch
d exe=/usr/sbin/ovs-vswitchd subj=system_u:system_r:openvswitch_t:s0 key=(null) 
type=AVC msg=audit(10/18/2017 23:04:18.563:142) : avc:  denied  { read write } for  pid=2624 comm=ovs-vswitchd name=tun dev="devtmpfs" ino=18627 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:tun_tap_device_t:s0 tclass=chr_file 
----
type=PROCTITLE msg=audit(10/18/2017 23:11:29.913:391) : proctitle=ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log 
type=SYSCALL msg=audit(10/18/2017 23:11:29.913:391) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x56497642d59c a1=O_RDWR a2=0x7fff3bc68a00 a3=0x8 items=0 ppid=4610 pid=4611 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ovs-vswitchd exe=/usr/sbin/ovs-vswitchd subj=system_u:system_r:openvswitch_t:s0 key=(null) 
type=AVC msg=audit(10/18/2017 23:11:29.913:391) : avc:  denied  { read write } for  pid=4611 comm=ovs-vswitchd name=vfio dev="devtmpfs" ino=18625 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:vfio_device_t:s0 tclass=chr_file
----

Comment 11 Christian Trautman 2017-11-27 20:36:10 UTC
As per Jeans recommendation I tried using a slightly later policy.

openvswitch-selinux-policy-2.8.90-2.fc25.noarch

This resolved the issue. I was able to start openvswitch with ports bound to vfio-pci. I also verified I could add a netdev bridge successfully.

I am still attaching the denials to the bug if you want to review them for the version that did not work.

With the working version I have no other need from this bug at this time.

Comment 12 Christian Trautman 2017-11-27 20:37:05 UTC
Created attachment 1359601 [details]
denials

Comment 13 Aaron Conole 2017-11-27 22:16:07 UTC
Given this was not tested previously, and that the package:

openstack-selinux (and dependent container-selinux) get past this as confirmed by Jean, I think this is not a blocker.

Comment 15 Jean-Tsung Hsiao 2018-01-26 18:39:51 UTC
Attached below is a brief status report of a Selinux/OVS testing under RHEL7.5 and RHEL7.4z.

*** Selinux/OVS 2.9 and  OVS 2.7 testing under RHEL7.5 ***

** Packages under test **

OVS --- OVS 2.9.0-0.3 fdP and OVS 2.7.3-3 fdP

Rhel7.5 --- RHEL-7.5-20180125.0/compose

Selinux --- selinux-policy-3.13.1-186

** Test results **

Without openstack-selinux and container-selinux packages installed, starting OVS with Selinux=Enforcing was successful and adding dpdk interfaces to OVS-dpdk bridge had no issues.

** Comments, Questions and Suggestions **

So, the key here is selinux-policy-3.13.1-186 --- no need to install openstack-selinux and container-selinux packages

*** Selinux/OVS 2.9 and OVS 2.7 testing under RHEL7.4z ***

** Packages under test **

OVS --- OVS 2.9.0-0.3 fdP and OVS 2.7.3-3 fdP

Rhel7.4z --- RHEL-7.4-updates-20180119.0/compose

Selinux --- selinux-policy-3.13.1-166

container-selinux --- container-selinux-2.41-1

openstack-selinux --- openstack-selinux-0.8.13-1

** Test results **

Without openstack-selinux and container-selinux packages, starting OVS with Selinux=Enforcing encountered AVC's and thus failed.

After installing openstack-selinux and container-selinux packages, starting OVS with Selinux=Enforcing was successful, and adding dpdk interfaces to OVS-dpdk bridge had no issues.

** Comments, Questions and Suggestions **

Can't upgrade selinux-policy to selinux-policy-3.13.1-186 due to many dependency issues.

So, is it possible to update the latest 7.4z compose so that selinux-policy-3.13.1-186 can be installed ?

At the mement need to add openstack-selinux and container-selinux packages to the compose so that OVS-dpdk can be run successfully with Selinux=Enforcing.

Comment 16 Jean-Tsung Hsiao 2018-01-26 20:32:26 UTC
An Update:

In a 7.5/OVS-2.7.3-3 fdP environment, encountered the following AVC's on starting an NFV guest:

type=AVC msg=audit(1516997142.618:388): avc:  denied  { read write } for  pid=2788 comm="vhost_thread1" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F342D6D715F7668755F67756573742F71656D755F6261636B5F6D656D2E5F6F626A656374735F72616D2D6E6F6465302E43486A656F72202864656C6574656429 dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file
type=AVC msg=audit(1516997142.627:389): avc:  denied  { read write } for  pid=2788 comm="vhost_thread1" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F342D6D715F7668755F67756573742F71656D755F6261636B5F6D656D2E5F6F626A656374735F72616D2D6E6F6465302E43486A656F72202864656C6574656429 dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file

This resulted in PvP test failure --- traffic from Xena traffic generator not going through guest testpmd.

Comment 17 Milos Malik 2018-01-30 10:59:20 UTC
Used AVCs from comment#16 and piped them into ausearch -i:
----
type=AVC msg=audit(01/26/2018 15:05:42.618:388) : avc:  denied  { read write } for  pid=2788 comm=vhost_thread1 path=/dev/hugepages/libvirt/qemu/4-mq_vhu_guest/qemu_back_mem._objects_ram-node0.CHjeor (deleted) dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file 
----
type=AVC msg=audit(01/26/2018 15:05:42.627:389) : avc:  denied  { read write } for  pid=2788 comm=vhost_thread1 path=/dev/hugepages/libvirt/qemu/4-mq_vhu_guest/qemu_back_mem._objects_ram-node0.CHjeor (deleted) dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file 
----

Comment 21 errata-xmlrpc 2018-10-30 10:01:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3111


Note You need to log in before you can comment on or make changes to this bug.