Bug 1262357 - ovs-vsctl add-br hangs - selinux denial
ovs-vsctl add-br hangs - selinux denial
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch (Show other bugs)
7.2
Unspecified Unspecified
urgent Severity urgent
: rc
: ---
Assigned To: Flavio Leitner
Network QE
: AutomationBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-11 09:26 EDT by Pavel Sedlák
Modified: 2015-10-28 23:40 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-28 23:40:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
strace of ovs-vsctl add-br psedlak-bridge (16.90 KB, application/x-bzip)
2015-09-11 09:28 EDT, Pavel Sedlák
no flags Details
strace of ovsdb-server service (6.52 KB, application/x-bzip)
2015-09-11 09:28 EDT, Pavel Sedlák
no flags Details

  None (edit)
Description Pavel Sedlák 2015-09-11 09:26:50 EDT
Created attachment 1072560 [details]
/var/log/audit/audit.log

Call to 'ovs-vsctl add-br psedlak-bridge' hangs indefinitely.
After interrupting and issuing ovs-vsctl list-br it shows the bridge there.

Based on strace (attached) seems it recieves some reply from ovsdb-server
but still continues to wait for more from the socket.

There appears to be some AVC Denials:
> type=AVC msg=audit(1441974588.740:536): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441974588.740:536): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd6705deb0 a1=80000 a2=7ffd6705dee6 a3=f items=0 ppid=4107 pid=4132 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441974588.740:537): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.dep.bin" dev="vda1" ino=17831792 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441974588.740:537): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd6705df10 a1=80000 a2=1a07130 a3=3 items=0 ppid=4107 pid=4132 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441974588.740:538): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.dep.bin" dev="vda1" ino=17831792 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441974588.740:538): arch=c000003e syscall=2 success=no exit=-13 a0=7ffd6705ce30 a1=80000 a2=1b6 a3=24 items=0 ppid=4107 pid=4132 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441974588.740:539): avc:  denied  { read } for  pid=4132 comm="modprobe" name="modules.alias.bin" dev="vda1" ino=17831794 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> ...
> type=AVC msg=audit(1441977010.107:803): avc:  denied  { read } for  pid=14700 comm="modprobe" name="modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=AVC msg=audit(1441977010.107:803): avc:  denied  { open } for  pid=14700 comm="modprobe" path="/usr/lib/modules/3.10.0-306.0.1.el7.x86_64/modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
> type=SYSCALL msg=audit(1441977010.107:803): arch=c000003e syscall=2 success=yes exit=3 a0=7ffca6d49e10 a1=80000 a2=7ffca6d49e46 a3=f items=0 ppid=14676 pid=14700 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
> type=AVC msg=audit(1441977010.108:804): avc:  denied  { getattr } for  pid=14700 comm="modprobe" path="/usr/lib/modules/3.10.0-306.0.1.el7.x86_64/modules.softdep" dev="vda1" ino=17831795 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file
>
> cat /var/log/audit/audit.log | audit2allow
> #============= openvswitch_t ==============
> allow openvswitch_t modules_dep_t:file { read getattr open };

With selinux disabled it seems to work!
> # setenforce 0
> # ovs-vsctl add-br psedlak-another
> # ovs-vsctl list-br
> psedlak-another
> psedlak-bridge

On rhel-7.2 (rhel-guest-image-7.2-20150821.0)
with openstack repos (using rhos-release tool)


yum info openvswitch

> Loaded plugins: search-disabled-repos
> Installed Packages
> Name        : openvswitch
> Arch        : x86_64
> Version     : 2.3.2
> Release     : 1.git20150730.el7_1
> Size        : 7.6 M
> Repo        : installed
> From repo   : rhelosp-7.0-puddle

repo info
> Repo-id      : rhelosp-7.0-puddle/x86_64
> Repo-name    : RHOS-7.0
> Repo-revision: 1441825843
> Repo-updated : Wed Sep  9 15:10:51 2015
> Repo-pkgs    : 495
> Repo-size    : 221 M
> Repo-baseurl : http:// <snip> rel-eng/OpenStack/7.0-RHEL-7/latest/RH7-RHOS-7.0/x86_64/os
> Repo-expire  : 21,600 second(s) (last: Fri Sep 11 08:28:12 2015)
> Repo-filename: /etc/yum.repos.d/rhos-release-7-rhel-7.2.repo

Strace output done like
> strace -v -ttt -s 2048 -Ff -o ovs-vsctl.trace ovs-vsctl add-br psedlak-bridge
for vsctl nad server attached though I guess those won't be needed if it's selinux.
Comment 1 Pavel Sedlák 2015-09-11 09:28:20 EDT
Created attachment 1072562 [details]
strace of ovs-vsctl add-br psedlak-bridge
Comment 2 Pavel Sedlák 2015-09-11 09:28:56 EDT
Created attachment 1072563 [details]
strace of ovsdb-server service
Comment 5 Flavio Leitner 2015-09-23 09:44:09 EDT
Looks like you haven't started the openvswitch service.
Could you do that before run any OVS command?
# systemctl start openvswitch

You can also enable it to start every boot.
# systemctl enable openvswitch


Thanks
fbl
Comment 10 Rashid Khan 2015-10-12 13:46:24 EDT
Hi Pavel, 
Please provide the information request above. 
If this is a real blocker, then we need the info from you to be able to unblock you.
If we do not get the required information in the next 5 business days we will assume it is NOT blocking you. 

thanks
Comment 11 Pavel Sedlák 2015-10-14 12:09:55 EDT
So I can confirm, that when running 'systemctl start openvswitch' first it works.
Though i have to add, that in case i run 'add-br/list-br' before systemct start,
i'm getting completely different behavior/output to the one descibed in my first message:
> ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
based on which i would guess it's not the only thing here (like that packstack/puppet would just not started the service).

I was running it now with OpenVSwitch Preview
>  2.4.0-1.el7                  @rhelosp-rhel-7-openvswitch
but then retried also with
>  2.3.2-1.git20150730.el7_1    rhelosp-7.0-puddle
too, both giving me same result (no db.sock when tried before systemctl start, no audit/denied issues at all)

# rpm -qa | grep selinux
> libselinux-2.2.2-6.el7.x86_64
> selinux-policy-targeted-3.13.1-53.el7.noarch
> libselinux-utils-2.2.2-6.el7.x86_64
> libselinux-python-2.2.2-6.el7.x86_64
> selinux-policy-3.13.1-53.el7.noarch

Though now newer guest image was used, and there are also additional rpm updates available (incl. kernel and selinux pkgs), so seems the possibility of version-combinations to try is quite big.

I'll retry tomorrow to run the whole openstack packstack installation/tests without workaround (which is injecting selinux module to allow modules_dep_t / openvswitch_t / read,getattr,open ... as obtained by audit2allow) to see if it works now.
Comment 13 Flavio Leitner 2015-10-16 19:35:24 EDT
(In reply to Pavel Sedlák from comment #11)
> So I can confirm, that when running 'systemctl start openvswitch' first it
> works.

Excellent.

> Though i have to add, that in case i run 'add-br/list-br' before systemct
> start,
> i'm getting completely different behavior/output to the one descibed in my
> first message:
> > ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

That is actually the correct behavior.  Without the daemon running, the db.sock shouldn't exist and the commands can't communicate.

It seems you had a partial service running.  I mean, daemons running without the kernel module, which is possible specially when using the userspace datapath, but then it suggests a problem elsewhere because by default the module is always loaded when the service starts.

> based on which i would guess it's not the only thing here (like that
> packstack/puppet would just not started the service).

Yup, sounds like the module was prevented to be loaded in the first place when the service was started and I can't tell the reason.

> I was running it now with OpenVSwitch Preview
> >  2.4.0-1.el7                  @rhelosp-rhel-7-openvswitch
> but then retried also with
> >  2.3.2-1.git20150730.el7_1    rhelosp-7.0-puddle
> too, both giving me same result (no db.sock when tried before systemctl
> start, no audit/denied issues at all)

Ok.

> Though now newer guest image was used, and there are also additional rpm
> updates available (incl. kernel and selinux pkgs), so seems the possibility
> of version-combinations to try is quite big.

We recently changed selinux policy to allow additional controller ports, but that doesn't impact the kernel module.

> I'll retry tomorrow to run the whole openstack packstack installation/tests
> without workaround (which is injecting selinux module to allow modules_dep_t
> / openvswitch_t / read,getattr,open ... as obtained by audit2allow) to see
> if it works now.

Ok, please keep us updated!
Thanks
fbl
Comment 14 Flavio Leitner 2015-10-26 12:44:10 EDT
Hello,
Any news here?
Thanks
Comment 15 Flavio Leitner 2015-10-28 23:20:53 EDT
I found this while looking for possible related updates in upstream/Fedora:

# rpm -qi selinux-policy-targeted
Name        : selinux-policy-targeted
Version     : 3.13.1
Release     : 128.13.fc22

ChangeLog:
[...]
* Mon Sep 14 2015 Lukas Vrabec <lvrabec@redhat.com> 3.13.1-128.13
- Allow openvswitch_t domains read kernel dependencies due to openvswitch run modprobe

Looks like there was an issue there, then looking back to RHEL-7 package:

commit 10c729353709901b9ea59f6b87113fe1268a5fe5
Author: Lukas Vrabec <lvrabec@redhat.com>
Date:   Thu Sep 10 15:25:43 2015 +0200

    * Thu Sep 10 2015 Lukas Vrabec <lvrabec@redhat.com> 3.13.1-48
[...]
    - Allow openvswitch_t domains read kernel dependencies due to openvswitch run modprobe

Maybe the image tested was a 7.2 devel image using an older package? At least the guest image is from Aug and the fix is from Sept.

The second attempt was made with selinux-policy -53 which contains the fix above.

You may try with selinux-policy -47 to reproduce and update to the latest to confirm the fix.
Comment 16 Flavio Leitner 2015-10-28 23:23:24 EDT
I am trying the version without the fix:

# rpm -q selinux-policy
selinux-policy-3.13.1-47.el7.noarch


type=AVC msg=audit(1446088788.733:1173): avc:  denied  { read } for  pid=23956 comm="modprobe" name="modules.dep.bin" dev="sda7" ino=800080 scontext=system_u:system_r:openvswitch_t:s0 tcontext=unconfined_u:object_r:modules_dep_t:s0 tclass=file
type=SYSCALL msg=audit(1446088788.733:1173): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe6fdeaa50 a1=80000 a2=bb02d0 a3=b items=0 ppid=23932 pid=23956 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1446088788.733:1174): avc:  denied  { read } for  pid=23956 comm="modprobe" name="modules.dep.bin" dev="sda7" ino=800080 scontext=system_u:system_r:openvswitch_t:s0 tcontext=unconfined_u:object_r:modules_dep_t:s0 tclass=file
type=SYSCALL msg=audit(1446088788.733:1174): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe6fde9960 a1=80000 a2=1b6 a3=3 items=0 ppid=23932 pid=23956 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)
type=AVC msg=audit(1446088788.733:1175): avc:  denied  { read } for  pid=23956 comm="modprobe" name="modules.alias.bin" dev="sda7" ino=800084 scontext=system_u:system_r:openvswitch_t:s0 tcontext=unconfined_u:object_r:modules_dep_t:s0 tclass=file
type=SYSCALL msg=audit(1446088788.733:1175): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe6fde9990 a1=80000 a2=1b6 a3=1 items=0 ppid=23932 pid=23956 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="modprobe" exe="/usr/bin/kmod" subj=system_u:system_r:openvswitch_t:s0 key=(null)


Looks like I have finally reproduced the issue.
Comment 17 Flavio Leitner 2015-10-28 23:27:35 EDT
And trying with:

# rpm -q selinux-policy
selinux-policy-3.13.1-48.el7.noarch

I can't reproduce the issue as expected.
Comment 18 Flavio Leitner 2015-10-28 23:40:03 EDT
The fix is part of RHEL-7.2 snapshot 5
http://download.devel.redhat.com/rel-eng/RHEL-7.2-Snapshot-5.0/compose/Server/x86_64/os/Packages/selinux-policy-3.13.1-60.el7.noarch.rpm

I am going to close this one.
If you need anything else, feel free to re-open it.
Thanks,
fbl

Note You need to log in before you can comment on or make changes to this bug.