RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1503835 - Openvswitch crash loop when adding netdev bridge ovs 2.7.2.10 FDP and selinux enabled
Summary: Openvswitch crash loop when adding netdev bridge ovs 2.7.2.10 FDP and selinux...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy
Version: 7.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Lukas Vrabec
QA Contact: Milos Malik
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-18 21:13 UTC by Christian Trautman
Modified: 2018-10-30 10:02 UTC (History)
12 users (show)

Fixed In Version: selinux-policy-3.13.1-197.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 10:01:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
SOSReport (12.78 MB, application/x-xz)
2017-10-18 21:13 UTC, Christian Trautman 		no flags Details
denials (5.76 MB, text/plain)
2017-11-27 20:37 UTC, Christian Trautman 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3111 0 None None None 2018-10-30 10:02:38 UTC

Description Christian Trautman 2017-10-18 21:13:28 UTC 
Created attachment 1340384 [details]
SOSReport

Description of problem:Adding a netdev bridge to Openvswitch with selinux enabled causes Openvswitch to enter into a crash loop.

2017-10-18T21:08:17.540Z|00017|dpdk|EMER|Requested device 0000:03:00.0 cannot be used
2017-10-18T21:08:17.883Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2017-10-18T21:08:17.887Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0
2017-10-18T21:08:17.887Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1
2017-10-18T21:08:17.887Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores
2017-10-18T21:08:17.887Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2017-10-18T21:08:17.888Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2017-10-18T21:08:17.889Z|00007|dpdk|INFO|DPDK Enabled - initializing...
2017-10-18T21:08:17.889Z|00008|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch
2017-10-18T21:08:17.889Z|00009|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 1024,1024 -c 0x00000001
2017-10-18T21:08:17.890Z|00010|dpdk|INFO|EAL: Detected 48 lcore(s)
2017-10-18T21:08:17.913Z|00011|dpdk|INFO|EAL: Probing VFIO support...
2017-10-18T21:08:17.913Z|00012|dpdk|ERR|EAL:   cannot open VFIO container, error 13 (Permission denied)
2017-10-18T21:08:17.913Z|00013|dpdk|INFO|EAL: VFIO support could not be initialized
2017-10-18T21:08:25.543Z|00014|dpdk|INFO|EAL: PCI device 0000:03:00.0 on NUMA socket 0
2017-10-18T21:08:25.543Z|00015|dpdk|INFO|EAL:   probe driver: 8086:10fb net_ixgbe
2017-10-18T21:08:25.544Z|00016|dpdk|EMER|EAL: Error - exiting with code: 1
  Cause: 



Version-Release number of selected component (if applicable):
RHEL 7.4 3.10.0-693.5.2.el7.x86_64
openvswitch openvswitch-2.7.2-10.git20170914.el7fdp.x86_64.rpm

How reproducible:always


Steps to Reproduce:
1. Install openvswitch
2. Start openvswitch service
3. Bind a card using driverctl to vfio-pci (ixgbe card is fine)
4. Add a netdev bridge



Actual results: Openvswitch enters reboot cycle


Expected results: Openvswitch does not restart


Additional info:
SOS report attached
Comment 2 Aaron Conole 2017-10-19 13:35:51 UTC 
From audit.log:

type=AVC msg=audit(1508361089.913:391): avc:  denied  { read write } for  pid=4611 comm="ovs-vswitchd" name="vfio" dev="devtmpfs" ino=18625 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:vfio_device_t:s0 tclass=chr_file

This should be fixed whenever bz 1482682 is addressed (I think).  A good test would be to take the selinux policy from upstream and test with it.
Comment 3 Christian Trautman 2017-10-19 15:07:26 UTC 
I will attempt to use the upstream policy and try it.
Comment 4 Jean-Tsung Hsiao 2017-11-15 01:07:42 UTC 
(In reply to Aaron Conole from comment #2)
> From audit.log:
> 
> type=AVC msg=audit(1508361089.913:391): avc:  denied  { read write } for 
> pid=4611 comm="ovs-vswitchd" name="vfio" dev="devtmpfs" ino=18625
> scontext=system_u:system_r:openvswitch_t:s0
> tcontext=system_u:object_r:vfio_device_t:s0 tclass=chr_file
> 
> This should be fixed whenever bz 1482682 is addressed (I think).  A good
> test would be to take the selinux policy from upstream and test with it.

Hi Aaron,

Bz 1482682 is for OVS-2.8. So, do you expect the new selinux policy can fix this issue.

Thanks!

Jean
Comment 5 Aaron Conole 2017-11-20 16:42:44 UTC 
The policy should still work even for 2.7.  That said, I'm concerned.  Do you see this issue with OvS 2.6, also?  I would expect yes.
Comment 6 Christian Trautman 2017-11-20 19:04:03 UTC 
Yes but the issue is different with 2.6. I tried version openvswitch-2.6.1-16.git20161206.el7ost.x86_64 and it just doesn't start at all with no bridges even created. 

It just enters a crash loop from failure to access the conf.db file.

Nov 20 13:58:38 localhost systemd: Starting Open vSwitch Database Unit...
Nov 20 13:58:38 localhost ovs-ctl: Starting ovsdb-server ovsdb-server: I/O error: /etc/openvswitch/conf.db: failed to lock lockfile (Resource temporarily unavailable)
Nov 20 13:58:38 localhost ovs-ctl: [FAILED]
Nov 20 13:58:38 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.14.0
Nov 20 13:58:38 localhost ovs-vsctl: ovs|00002|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Nov 20 13:58:38 localhost ovs-ctl: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
Nov 20 13:58:38 localhost systemd: ovsdb-server.service: control process exited, code=exited status=1
Nov 20 13:58:38 localhost systemd: Failed to start Open vSwitch Database Unit.
Nov 20 13:58:38 localhost systemd: Unit ovsdb-server.service entered failed state.
Nov 20 13:58:38 localhost systemd: ovsdb-server.service failed.
Nov 20 13:58:38 localhost systemd: ovsdb-server.service holdoff time over, scheduling restart.

I will try your instructions to enable the policy and get back to you with that info.
Comment 7 Christian Trautman 2017-11-22 15:18:42 UTC 
Hi Aaron,

I applied the upstream policy as per your instructions.

rpm -qa

openvswitch-2.7.2-10.git20170914.el7fdp.x86_64
openvswitch-selinux-policy-2.8.90-1.el7.noarch

I then tried to bind two devices to vfio-pci and start up openvswitch. It is still in a crash loop.
Comment 8 Jean-Tsung Hsiao 2017-11-23 04:05:09 UTC 
(In reply to Christian Trautman from comment #7)
> Hi Aaron,
> 
> I applied the upstream policy as per your instructions.
> 
> rpm -qa
> 
> openvswitch-2.7.2-10.git20170914.el7fdp.x86_64
> openvswitch-selinux-policy-2.8.90-1.el7.noarch
> 
> I then tried to bind two devices to vfio-pci and start up openvswitch. It is
> still in a crash loop.

Hi Christ,

The openvswitch-selinux-policy package makes the difference. Below I am using a newer version, and the issue is gone.

Thanks!

Jean

[root@netqe5 ovs-2.7.2-10-testing]# rpm -q openvswitch-selinux-policy
openvswitch-selinux-policy-2.8.90-2.fc25.noarch

[root@netqe5 ovs-2.7.2-10-testing]# getenforce
Enforcing
[root@netqe5 ovs-2.7.2-10-testing]# vs
290c3841-84ea-4aa3-b259-ea43a7a5b344
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "dpdk-10"
            Interface "dpdk-10"
                type: dpdk
                options: {dpdk-devargs="0000:81:00.0", n_rxq="2"}
        Port "dpdk-11"
            Interface "dpdk-11"
                type: dpdk
                options: {dpdk-devargs="0000:81:00.1", n_rxq="2"}
    ovs_version: "2.7.2"
[root@netqe5 ovs-2.7.2-10-testing]# !ps
ps -elf | grep ovs-vs
5 S root     134931      1 99  70 -10 - 1924168 poll_s 22:54 ?      00:12:17 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
0 S root     135065 120397  0  80   0 - 28165 pipe_w 22:58 pts/0    00:00:00 grep --color=auto ovs-vs
[root@netqe5 ovs-2.7.2-10-testing]# ps -elf | grep ovs-vs
5 S root     134931      1 99  70 -10 - 1924168 poll_s 22:54 ?      00:13:12 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach
0 S root     135067 120397  0  80   0 - 28165 pipe_w 22:58 pts/0    00:00:00 grep --color=auto ovs-vs
Comment 9 Milos Malik 2017-11-27 06:59:04 UTC 
Could you collect all SELinux denials from the machine and attach them here?

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today
Comment 10 Milos Malik 2017-11-27 07:08:26 UTC 
Based on the attached sosreport, following policy rules are needed:

allow openvswitch_t tun_tap_device_t:chr_file { read write };
allow openvswitch_t vfio_device_t:chr_file { read write };
allow virtlogd_t svirt_image_t:file unlink;
allow virtlogd_t virt_tmp_t:file unlink;

Here are the SELinux denials in raw form:
----
type=PROCTITLE msg=audit(10/18/2017 16:26:26.685:219) : proctitle=/usr/sbin/virtlogd 
type=SYSCALL msg=audit(10/18/2017 16:26:26.685:219) : arch=x86_64 syscall=unlink success=yes exit=0 a0=0x7f57dc000ce0 a1=0x7f57dc000934 a2=0x0 a3=0x2 items=0 ppid=1 pid=26050 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(10/18/2017 16:26:26.685:219) : avc:  denied  { unlink } for  pid=26050 comm=virtlogd name=master.console dev="dm-0" ino=735292 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:virt_tmp_t:s0 tclass=file
----
type=PROCTITLE msg=audit(10/18/2017 16:31:03.394:252) : proctitle=/usr/sbin/virtlogd 
type=SYSCALL msg=audit(10/18/2017 16:31:03.394:252) : arch=x86_64 syscall=unlink success=yes exit=0 a0=0x
7f57dc000ca0 a1=0x7f57dc0008d4 a2=0x0 a3=0x2 items=0 ppid=1 pid=26050 auid=unset uid=root gid=root euid=r
oot suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=virtlogd exe=/usr/sbin/
virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(10/18/2017 16:31:03.394:252) : avc:  denied  { unlink } for  pid=26050 comm=virtlogd name=master.console dev="dm-0" ino=902657 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:svirt_image_t:s0:c636,c936 tclass=file 
----
type=PROCTITLE msg=audit(10/18/2017 23:04:18.563:142) : proctitle=ovs-vswitchd unix:/var/run/openvswitch/
db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log 
type=SYSCALL msg=audit(10/18/2017 23:04:18.563:142) : arch=x86_64 syscall=open success=no exit=EACCES(Per
mission denied) a0=0x55fe00bd8d00 a1=O_RDWR a2=0x0 a3=0x3 items=0 ppid=1 pid=2624 auid=unset uid=root gid
=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ovs-vswitch
d exe=/usr/sbin/ovs-vswitchd subj=system_u:system_r:openvswitch_t:s0 key=(null) 
type=AVC msg=audit(10/18/2017 23:04:18.563:142) : avc:  denied  { read write } for  pid=2624 comm=ovs-vswitchd name=tun dev="devtmpfs" ino=18627 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:tun_tap_device_t:s0 tclass=chr_file 
----
type=PROCTITLE msg=audit(10/18/2017 23:11:29.913:391) : proctitle=ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log 
type=SYSCALL msg=audit(10/18/2017 23:11:29.913:391) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x56497642d59c a1=O_RDWR a2=0x7fff3bc68a00 a3=0x8 items=0 ppid=4610 pid=4611 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ovs-vswitchd exe=/usr/sbin/ovs-vswitchd subj=system_u:system_r:openvswitch_t:s0 key=(null) 
type=AVC msg=audit(10/18/2017 23:11:29.913:391) : avc:  denied  { read write } for  pid=4611 comm=ovs-vswitchd name=vfio dev="devtmpfs" ino=18625 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:vfio_device_t:s0 tclass=chr_file
----
Comment 11 Christian Trautman 2017-11-27 20:36:10 UTC 
As per Jeans recommendation I tried using a slightly later policy.

openvswitch-selinux-policy-2.8.90-2.fc25.noarch

This resolved the issue. I was able to start openvswitch with ports bound to vfio-pci. I also verified I could add a netdev bridge successfully.

I am still attaching the denials to the bug if you want to review them for the version that did not work.

With the working version I have no other need from this bug at this time.
Comment 12 Christian Trautman 2017-11-27 20:37:05 UTC 
Created attachment 1359601 [details]
denials
Comment 13 Aaron Conole 2017-11-27 22:16:07 UTC 
Given this was not tested previously, and that the package:

openstack-selinux (and dependent container-selinux) get past this as confirmed by Jean, I think this is not a blocker.
Comment 15 Jean-Tsung Hsiao 2018-01-26 18:39:51 UTC 
Attached below is a brief status report of a Selinux/OVS testing under RHEL7.5 and RHEL7.4z.

*** Selinux/OVS 2.9 and  OVS 2.7 testing under RHEL7.5 ***

** Packages under test **

OVS --- OVS 2.9.0-0.3 fdP and OVS 2.7.3-3 fdP

Rhel7.5 --- RHEL-7.5-20180125.0/compose

Selinux --- selinux-policy-3.13.1-186

** Test results **

Without openstack-selinux and container-selinux packages installed, starting OVS with Selinux=Enforcing was successful and adding dpdk interfaces to OVS-dpdk bridge had no issues.

** Comments, Questions and Suggestions **

So, the key here is selinux-policy-3.13.1-186 --- no need to install openstack-selinux and container-selinux packages

*** Selinux/OVS 2.9 and OVS 2.7 testing under RHEL7.4z ***

** Packages under test **

OVS --- OVS 2.9.0-0.3 fdP and OVS 2.7.3-3 fdP

Rhel7.4z --- RHEL-7.4-updates-20180119.0/compose

Selinux --- selinux-policy-3.13.1-166

container-selinux --- container-selinux-2.41-1

openstack-selinux --- openstack-selinux-0.8.13-1

** Test results **

Without openstack-selinux and container-selinux packages, starting OVS with Selinux=Enforcing encountered AVC's and thus failed.

After installing openstack-selinux and container-selinux packages, starting OVS with Selinux=Enforcing was successful, and adding dpdk interfaces to OVS-dpdk bridge had no issues.

** Comments, Questions and Suggestions **

Can't upgrade selinux-policy to selinux-policy-3.13.1-186 due to many dependency issues.

So, is it possible to update the latest 7.4z compose so that selinux-policy-3.13.1-186 can be installed ?

At the mement need to add openstack-selinux and container-selinux packages to the compose so that OVS-dpdk can be run successfully with Selinux=Enforcing.
Comment 16 Jean-Tsung Hsiao 2018-01-26 20:32:26 UTC 
An Update:

In a 7.5/OVS-2.7.3-3 fdP environment, encountered the following AVC's on starting an NFV guest:

type=AVC msg=audit(1516997142.618:388): avc:  denied  { read write } for  pid=2788 comm="vhost_thread1" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F342D6D715F7668755F67756573742F71656D755F6261636B5F6D656D2E5F6F626A656374735F72616D2D6E6F6465302E43486A656F72202864656C6574656429 dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file
type=AVC msg=audit(1516997142.627:389): avc:  denied  { read write } for  pid=2788 comm="vhost_thread1" path=2F6465762F6875676570616765732F6C6962766972742F71656D752F342D6D715F7668755F67756573742F71656D755F6261636B5F6D656D2E5F6F626A656374735F72616D2D6E6F6465302E43486A656F72202864656C6574656429 dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file

This resulted in PvP test failure --- traffic from Xena traffic generator not going through guest testpmd.
Comment 17 Milos Malik 2018-01-30 10:59:20 UTC 
Used AVCs from comment#16 and piped them into ausearch -i:
----
type=AVC msg=audit(01/26/2018 15:05:42.618:388) : avc:  denied  { read write } for  pid=2788 comm=vhost_thread1 path=/dev/hugepages/libvirt/qemu/4-mq_vhu_guest/qemu_back_mem._objects_ram-node0.CHjeor (deleted) dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file 
----
type=AVC msg=audit(01/26/2018 15:05:42.627:389) : avc:  denied  { read write } for  pid=2788 comm=vhost_thread1 path=/dev/hugepages/libvirt/qemu/4-mq_vhu_guest/qemu_back_mem._objects_ram-node0.CHjeor (deleted) dev="hugetlbfs" ino=107915 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:svirt_image_t:s0 tclass=file 
----
Comment 21 errata-xmlrpc 2018-10-30 10:01:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3111
Note You need to log in before you can comment on or make changes to this bug.