RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1262357 - ovs-vsctl add-br hangs - selinux denial
Summary: ovs-vsctl add-br hangs - selinux denial
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch
Version: 7.2
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Flavio Leitner
QA Contact: Network QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-11 13:26 UTC by Pavel Sedlák
Modified: 2015-10-29 03:40 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-29 03:40:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
strace of ovs-vsctl add-br psedlak-bridge (16.90 KB, application/x-bzip)
2015-09-11 13:28 UTC, Pavel Sedlák
no flags Details
strace of ovsdb-server service (6.52 KB, application/x-bzip)
2015-09-11 13:28 UTC, Pavel Sedlák
no flags Details

Description Pavel Sedlák 2015-09-11 13:26:50 UTC
Created attachment 1072560 [details]
/var/log/audit/audit.log

Call to 'ovs-vsctl add-br psedlak-bridge' hangs indefinitely.
After interrupting and issuing ovs-vsctl list-br it shows the bridge there.

Based on strace (attached) seems it recieves some reply from ovsdb-server
but still continues to wait for more from the socket.

There appears to be some AVC Denials:
> type=AVC msg=audit(1441974588.740:536): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441974588.740:536): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd6705deb0 a1=80000 a2=7ffd6705dee6 a3=f items=0 ppid=4107 pid=4132 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441974588.740:537): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.dep.bin" dev="vda1" ino=17831792 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441974588.740:537): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd6705df10 a1=80000 a2=1a07130 a3=3 items=0 ppid=4107 pid=4132 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441974588.740:538): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.dep.bin" dev="vda1" ino=17831792 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441974588.740:538): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd6705ce30 a1=80000 a2=1b6 a3=24 items=0 ppid=4107 pid=4132 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441974588.740:539): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.alias.bin" dev="vda1" ino=17831794 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> ...
> type=AVC msg=audit(1441977010.107:803): avc:  denied  { read } for  pid=14700 comm="modprobe" name="modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=AVC msg=audit(1441977010.107:803): avc:  denied  { open } for  pid=14700 comm="modprobe" path="/usr/lib/modules/3.10.0-306.0.1.el7.x86_64/modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441977010.107:803): arch=c000003e syscall=2 success=yes exit=3 a0=7ffca6d49e10 a1=80000 a2=7ffca6d49e46 a3=f items=0 ppid=14676 pid=14700 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441977010.108:804): avc:  denied  { getattr } for  pid=14700 comm="modprobe" path="/usr/lib/modules/3.10.0-306.0.1.el7.x86_64/modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
>
> cat /var/log/audit/audit.log | audit2allow
> #============= openvswitch_t ==============
> allow openvswitch_t modules_dep_t:file { read getattr open };

With selinux disabled it seems to work!
> # setenforce 0
> # ovs-vsctl add-br psedlak-another
> # ovs-vsctl list-br
> psedlak-another
> psedlak-bridge

On rhel-7.2 (rhel-guest-image-7.2-20150821.0)
with openstack repos (using rhos-release tool)


yum info openvswitch

> Loaded plugins: search-disabled-repos
> Installed Packages
> Name        : openvswitch
> Arch        : x86_64
> Version     : 2.3.2
> Release     : 1.git20150730.el7_1
> Size        : 7.6 M
> Repo        : installed
> From repo   : rhelosp-7.0-puddle

repo info
> Repo-id      : rhelosp-7.0-puddle/x86_64
> Repo-name    : RHOS-7.0
> Repo-revision: 1441825843
> Repo-updated : Wed Sep  9 15:10:51 2015
> Repo-pkgs    : 495
> Repo-size    : 221 M
> Repo-baseurl : http:// <snip> rel-eng/OpenStack/7.0-RHEL-7/latest/RH7-RHOS-7.0/x86_64/os
> Repo-expire  : 21,600 second(s) (last: Fri Sep 11 08:28:12 2015)
> Repo-filename: /etc/yum.repos.d/rhos-release-7-rhel-7.2.repo

Strace output done like
> strace -v -ttt -s 2048 -Ff -o ovs-vsctl.trace ovs-vsctl add-br psedlak-bridge
for vsctl nad server attached though I guess those won't be needed if it's selinux.

Comment 1 Pavel Sedlák 2015-09-11 13:28:20 UTC
Created attachment 1072562 [details]
strace of ovs-vsctl add-br psedlak-bridge

Comment 2 Pavel Sedlák 2015-09-11 13:28:56 UTC
Created attachment 1072563 [details]
strace of ovsdb-server service

Comment 5 Flavio Leitner 2015-09-23 13:44:09 UTC
Looks like you haven't started the openvswitch service.
Could you do that before run any OVS command?
# systemctl start openvswitch

You can also enable it to start every boot.
# systemctl enable openvswitch


Thanks
fbl

Comment 10 Rashid Khan 2015-10-12 17:46:24 UTC
Hi Pavel, 
Please provide the information request above. 
If this is a real blocker, then we need the info from you to be able to unblock you.
If we do not get the required information in the next 5 business days we will assume it is NOT blocking you. 

thanks

Comment 11 Pavel Sedlák 2015-10-14 16:09:55 UTC
So I can confirm, that when running 'systemctl start openvswitch' first it works.
Though i have to add, that in case i run 'add-br/list-br' before systemct start,
i'm getting completely different behavior/output to the one descibed in my first message:
> ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
based on which i would guess it's not the only thing here (like that packstack/puppet would just not started the service).

I was running it now with OpenVSwitch Preview
>  2.4.0-1.el7                  @rhelosp-rhel-7-openvswitch
but then retried also with
>  2.3.2-1.git20150730.el7_1    rhelosp-7.0-puddle
too, both giving me same result (no db.sock when tried before systemctl start, no audit/denied issues at all)

# rpm -qa | grep selinux
> libselinux-2.2.2-6.el7.x86_64
> selinux-policy-targeted-3.13.1-53.el7.noarch
> libselinux-utils-2.2.2-6.el7.x86_64
> libselinux-python-2.2.2-6.el7.x86_64
> selinux-policy-3.13.1-53.el7.noarch

Though now newer guest image was used, and there are also additional rpm updates available (incl. kernel and selinux pkgs), so seems the possibility of version-combinations to try is quite big.

I'll retry tomorrow to run the whole openstack packstack installation/tests without workaround (which is injecting selinux module to allow modules_dep_t / openvswitch_t / read,getattr,open ... as obtained by audit2allow) to see if it works now.

Comment 13 Flavio Leitner 2015-10-16 23:35:24 UTC
(In reply to Pavel Sedlák from comment #11)
> So I can confirm, that when running 'systemctl start openvswitch' first it
> works.

Excellent.

> Though i have to add, that in case i run 'add-br/list-br' before systemct
> start,
> i'm getting completely different behavior/output to the one descibed in my
> first message:
> > ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

That is actually the correct behavior.  Without the daemon running, the db.sock shouldn't exist and the commands can't communicate.

It seems you had a partial service running.  I mean, daemons running without the kernel module, which is possible specially when using the userspace datapath, but then it suggests a problem elsewhere because by default the module is always loaded when the service starts.

> based on which i would guess it's not the only thing here (like that
> packstack/puppet would just not started the service).

Yup, sounds like the module was prevented to be loaded in the first place when the service was started and I can't tell the reason.

> I was running it now with OpenVSwitch Preview
> >  2.4.0-1.el7                  @rhelosp-rhel-7-openvswitch
> but then retried also with
> >  2.3.2-1.git20150730.el7_1    rhelosp-7.0-puddle
> too, both giving me same result (no db.sock when tried before systemctl
> start, no audit/denied issues at all)

Ok.

> Though now newer guest image was used, and there are also additional rpm
> updates available (incl. kernel and selinux pkgs), so seems the possibility
> of version-combinations to try is quite big.

We recently changed selinux policy to allow additional controller ports, but that doesn't impact the kernel module.

> I'll retry tomorrow to run the whole openstack packstack installation/tests
> without workaround (which is injecting selinux module to allow modules_dep_t
> / openvswitch_t / read,getattr,open ... as obtained by audit2allow) to see
> if it works now.

Ok, please keep us updated!
Thanks
fbl

Comment 14 Flavio Leitner 2015-10-26 16:44:10 UTC
Hello,
Any news here?
Thanks

Comment 15 Flavio Leitner 2015-10-29 03:20:53 UTC
I found this while looking for possible related updates in upstream/Fedora:

# rpm -qi selinux-policy-targeted
Name        : selinux-policy-targeted
Version     : 3.13.1
Release     : 128.13.fc22

ChangeLog:
[...]
* Mon Sep 14 2015 Lukas Vrabec <lvrabec> 3.13.1-128.13
- Allow openvswitch_t domains read kernel dependencies due to openvswitch run modprobe

Looks like there was an issue there, then looking back to RHEL-7 package:

commit 10c729353709901b9ea59f6b87113fe1268a5fe5
Author: Lukas Vrabec <lvrabec>
Date:   Thu Sep 10 15:25:43 2015 +0200

    * Thu Sep 10 2015 Lukas Vrabec <lvrabec> 3.13.1-48
[...]
    - Allow openvswitch_t domains read kernel dependencies due to openvswitch run modprobe

Maybe the image tested was a 7.2 devel image using an older package? At least the guest image is from Aug and the fix is from Sept.

The second attempt was made with selinux-policy -53 which contains the fix above.

You may try with selinux-policy -47 to reproduce and update to the latest to confirm the fix.

Comment 16 Flavio Leitner 2015-10-29 03:23:24 UTC
I am trying the version without the fix:

# rpm -q selinux-policy
selinux-policy-3.13.1-47.el7.noarch


type=AVC msg=audit(1446088788.733:1173): avc:  denied  { read } for  pid=23956 comm="modprobe" name="modules.dep.bin" dev="sda7" ino=800080 scontext=system_u:system_r:openvswitch_t:s0 tcontext=unconfined_u:object_r:modules_dep_t:s0 tclass=file
type=SYSCALL msg=audit(1446088788.733:1173): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe6fdeaa50 a1=80000 a2=bb02d0 a3=b items=0 ppid=23932 pid=23956 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1446088788.733:1174): avc:  denied  { read } for  pid=23956 comm="modprobe" name="modules.dep.bin" dev="sda7" ino=800080 scontext=system_u:system_r:openvswitch_t:s0 tcontext=unconfined_u:object_r:modules_dep_t:s0 tclass=file
type=SYSCALL msg=audit(1446088788.733:1174): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe6fde9960 a1=80000 a2=1b6 a3=3 items=0 ppid=23932 pid=23956 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1446088788.733:1175): avc:  denied  { read } for  pid=23956 comm="modprobe" name="modules.alias.bin" dev="sda7" ino=800084 scontext=system_u:system_r:openvswitch_t:s0 tcontext=unconfined_u:object_r:modules_dep_t:s0 tclass=file
type=SYSCALL msg=audit(1446088788.733:1175): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe6fde9990 a1=80000 a2=1b6 a3=1 items=0 ppid=23932 pid=23956 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)


Looks like I have finally reproduced the issue.

Comment 17 Flavio Leitner 2015-10-29 03:27:35 UTC
And trying with:

# rpm -q selinux-policy
selinux-policy-3.13.1-48.el7.noarch

I can't reproduce the issue as expected.

Comment 18 Flavio Leitner 2015-10-29 03:40:03 UTC
The fix is part of RHEL-7.2 snapshot 5
http://download.devel.redhat.com/rel-eng/RHEL-7.2-Snapshot-5.0/compose/Server/x86_64/os/Packages/selinux-policy-3.13.1-60.el7.noarch.rpm

I am going to close this one.
If you need anything else, feel free to re-open it.
Thanks,
fbl


Note You need to log in before you can comment on or make changes to this bug.