Bug 1503820

Summary: openvswitch fails to start: PAM audit_open() failed: Permission denied
Product: Red Hat Enterprise Linux 7 Reporter: Marcin Mirecki <mmirecki>
Component: openvswitchAssignee: Aaron Conole <aconole>
Status: CLOSED DUPLICATE QA Contact: Junhan <juyan>
Severity: high Docs Contact:
Priority: high    
Version: 7.6-AltCC: aconole, atragler, danken, dron, eedri, mkalfon, mmirecki, qding, rbost, rkhan
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-27 22:18:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1482682    
Bug Blocks: 1503566    

Description Marcin Mirecki 2017-10-18 20:33:53 UTC
openvswitch fails to start when selinux is enabled.
It fails to create the ovs db saying: PAM audit_open() failed: Permission denied

It works fine when selinux is disabled.

The problem occurs in build:
openvswitch-2.8.0-1.el7fdb
and the master branch (as of 2017-10-16).

Comment 2 Marcin Mirecki 2017-10-19 07:44:40 UTC
Additional info:

#ls -lah /etc/openvswitch
drwxr-xr-x.   2 root root 4.0K Oct 17 15:07 .
drwxr-xr-x. 100 root root 4.0K Oct 17 14:21 ..
-rw-r--r--.   1 root root  15K Oct 17 15:07 conf.db.backup-
-rw-------.   1 root root    0 Oct 17 14:56 .conf.db.~lock~
-rw-r--r--.   1 root root  163 Oct 17 14:08 default.conf
-rw-r--r--.   1 root root   37 Oct 17 14:56 system-id.conf

# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.8.90
DB Schema 7.15.1

Build from master at: commit 7468ec78853e4c82865776c6c003a64b7b8b2a5e
Build from source and installed on a fresh vm from the built rpms.
but also occurs when installed from brew build: openvswitch-2.8.0-1.el7fdb

journalctl says:

Oct 19 09:34:55 f25_56 audit[1450]: AVC avc:  denied  { create } for  pid=1450 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket p
Oct 19 09:34:55 f25_56 runuser[1450]: PAM audit_open() failed: Permission denied
Oct 19 09:34:55 f25_56 ovs-ctl[1419]: runuser: System error
Oct 19 09:34:55 f25_56 ovs-ctl[1419]: /etc/openvswitch/conf.db does not exist ... (warning).
Oct 19 09:34:55 f25_56 audit[1452]: AVC avc:  denied  { create } for  pid=1452 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket p
Oct 19 09:34:55 f25_56 runuser[1452]: PAM audit_open() failed: Permission denied
Oct 19 09:34:55 f25_56 ovs-ctl[1419]: Creating empty database /etc/openvswitch/conf.db runuser: System error
Oct 19 09:34:55 f25_56 ovs-ctl[1419]: [FAILED]
Oct 19 09:34:55 f25_56 systemd[1]: ovsdb-server.service: Control process exited, code=exited status=1
Oct 19 09:34:55 f25_56 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ovsdb-server comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? t
Oct 19 09:34:55 f25_56 systemd[1]: Failed to start Open vSwitch Database Unit.
-- Subject: Unit ovsdb-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ovsdb-server.service has failed.

Comment 3 Aaron Conole 2017-10-19 13:27:50 UTC
If you're building from master, you should also have an openvswitch-selinux-policy rpm.  Please install this - that should eliminate the selinux issues.  There is currently open bz 1482682 to track those changes.

I'm also curious why /etc/openvswitch and all it's parts are owned by root - was there additional configurations that you had?  What are the contents of /etc/sysconfig/openvswitch - my guess is you may have some kinds of permissions problems popping up with various install / remove / upgrade paths.

Comment 4 Aaron Conole 2017-10-20 19:30:28 UTC
Forgot to set the needinfo flag.  See comment #3.

Comment 6 Marcin Mirecki 2017-11-14 20:53:46 UTC
I've build master and installed openvswitch-selinux-policy-2.8.90-1.fc25.noarch.rpm
This indeed fixes the problem!

The content of the /etc/openvswitch/ and /etc/sysconfig/openvswitch dirs listed below. Keep in mind this is my dev environment, so it has quite a history of all kinds of tests and tweaks (although the problem also occured on our test envs, which are recreated from scratch every time).

# ls -al /etc/openvswitch/
total 8276
drwxr-xr-x.   2 root root    4096 Nov 14 21:46 .
drwxr-xr-x. 191 root root   12288 Nov 14 21:24 ..
-rw-------.   1 root root       0 Sep 29  2016 .conf.db.tmp.~lock~
-rw-------.   1 root root       0 Nov 14 21:46 .conf.db.~lock~
-rw-------.   1 root root       0 Aug 24  2016 .ovnnb.db.~lock~
-rw-------.   1 root root       0 Aug 17 09:14 .ovnnb_db.db.tmp.~lock~
-rw-------.   1 root root       0 Jan 19  2017 .ovnnb_db.db.~lock~
-rw-------.   1 root root       0 Aug 24  2016 .ovnsb.db.~lock~
-rw-------.   1 root root       0 Aug 17 09:14 .ovnsb_db.db.tmp.~lock~
-rw-------.   1 root root       0 Jan 19  2017 .ovnsb_db.db.~lock~
-rw-r--r--.   1 root root  218948 Nov  8 14:09 conf.db
-rw-r--r--.   1 root root   14185 Oct  3  2016 conf.db.backup7.13.0-2202834738
-rw-r--r--.   1 root root   16868 Aug  8 18:10 conf.db.backup7.14.0-3374030633
-rw-r--r--.   1 root root 7001822 Oct  5 13:06 conf.db.backup7.14.0-3974332717
-rw-r--r--.   1 root root     163 Nov 14 21:25 default.conf
-rw-r--r--.   1 root root   79997 Sep 29  2016 ovnnb.db
-rw-r--r--.   1 root root  150696 Oct  3 14:05 ovnnb_db.db
-rw-r--r--.   1 root root   18242 Aug 17 09:14 ovnnb_db.db.backup5.4.1-3485560318
-rw-r--r--.   1 root root  186385 Sep 29  2016 ovnsb.db
-rw-r--r--.   1 root root  504863 Oct  3 14:05 ovnsb_db.db
-rw-r--r--.   1 root root  220447 Aug 17 09:14 ovnsb_db.db.backup1.9.0-2240045372
-rw-r--r--.   1 root root      37 Sep 20  2016 system-id.conf

# ls -al /etc/sysconfig/openvswitch
-rw-r--r--. 1 root root 755 Nov 14 21:25 /etc/sysconfig/openvswitch

Comment 7 Marcin Mirecki 2017-11-17 08:53:18 UTC
I just noticed that none of the latest brew build include the openvswitch-selinux-policy package.
The latest one as of today is:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=623960

When could we expect a build that includes this:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=623960

Comment 8 Aaron Conole 2017-11-21 16:02:37 UTC
We don't ship our own selinux packages.  RHEL uses selinux-policy package.  That is being updated (see bz #1482682), but I'm not sure when it will land.

Is that all that is the problem?  If so, we can probably close this as duplicate of that?