Bug 1417164

Created attachment 1245101 [details]
sosreport Compute

Created attachment 1245102 [details]
sosreport controller

Created attachment 1245122 [details]
Traceback from OVS agent

I find that, after setting the SELINUX to permissive and reboot, it works.

Created attachment 1245154 [details]
SELINUX logs (/var/log/messages)

Switching component to SELinux. Since we want to update OVS to 2.6 in more than one OSP release stream we'll actually need to duplicate this bug once it's fixed and make sure it's fixed in likely at least OSP 8, 9, 10 and 11.

type=AVC msg=audit(1485511068.007:113): avc: denied { unlink } for pid=2319 comm="NetworkManager" name="dhclient-ens802f0.pid" dev="tmpfs" ino=16544 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file

/var/run/dhclient-eno1.pid should be labeled dhcpc_var_run_t not var_run_t.
Try `restorecon -Rv /var/run/dhclient-eno1.pid`.

This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Hmm, Ryan, there's something else going on here.

A similar issue happened recently with /var/run/haproxy.sock - the context is correct if you restorecon, but touching the file doesn't set it properly.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/SELinux_Users_and_Administrators_Guide/sect-Security-Enhanced_Linux-Working_with_SELinux-SELinux_Contexts_Labeling_Files.html

[root@localhost run]# touch /var/run/dhclient-eno1.pid
[root@localhost run]# ls -lZ !$
ls -lZ /var/run/dhclient-eno1.pid
-rw-r--r--. root root unconfined_u:object_r:var_run_t:s0 /var/run/dhclient-eno1.pid
[root@localhost run]# restorecon !$
restorecon /var/run/dhclient-eno1.pid
[root@localhost run]# ls -lZ !$
ls -lZ /var/run/dhclient-eno1.pid
-rw-r--r--. root root unconfined_u:object_r:dhcpc_var_run_t:s0 /var/run/dhclient-eno1.pid


So, the context is there, it's just not getting set quite correctly. Is something besides NetworkManager creating it?

I ask because other such files created by NetworkManager do have the right label on my box.

Hi Karthik, is this issue reproducible? We have a bug 1425507 with the same traceback so I  would like to ask you, if you can reproduce this and try out to:

1) Kill rootwrap daemon before starting OVS agent.

2) If that doesn't help: confirm that setting selinux to permissive solves it *without* rebooting the node. Just setenforce 0 and start agent (after you confirm that you can't start it).

We tried your steps and I have attached the logs - http://paste.openstack.org/show/603165/

After we disable the selinux with setenforce 0 and restarting agent, brings up the br-int interface.

Daniel will try to reproduce the bug tomorrow.

*** Bug 1433432 has been marked as a duplicate of this bug. ***

I'll update this tomorrow but I've seen this other bug [0] with the *exact* same denial message. Apparently they installed ovs manually and something went wrong during the installation, then they did the process again and it worked. 
I'll try to install the linked package in this RHBZ on an OSP10 all-in-one and see if it reproduces the issue.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1402032

Can I have access to this environment? It looks like the system is mislabeled so I'd want to restore context (sudo restorecon -RV /) and/or upgrade selinux-policy package to check if we still see denials on ovs after that.

Thanks guys for providing access to the environment. This is what i've seen so far:

- Both compute nodes had denials for ovs-vsctl:

time->Fri Mar 24 13:54:33 2017
type=SYSCALL msg=audit(1490363673.009:278): arch=c000003e syscall=21 success=no exit=-13 a0=7fc4a8000ce0 a1=1 a2=1 a3=fffff000 items=0 ppid=86487 pid=86493 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="neutron-rootwra" exe="/usr/bin/python2.7" subj=system_u:system_r:neutron_t:s0 key=(null)
type=AVC msg=audit(1490363673.009:278): avc:  denied  { execute } for  pid=86493 comm="neutron-rootwra" name="ovs-vsctl" dev="sdb2" ino=2139730 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file


- Apparently, ovs binaries are unlabeled:

[root@overcloud-compute-0 audit]# ls -Z /usr/bin/ovs-*
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-appctl
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-dpctl
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-dpctl-top
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-ofctl
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-pki
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-vsctl

- Running "restorecon -Rv /" in one of the nodes, solved the issue and, after restarting ovs agent, I can't see any more denials and files are correctly labeled:

[root@overcloud-compute-0 audit]# ls -Z /usr/bin/ovs-*
-rwxr-xr-x. root root system_u:object_r:openvswitch_exec_t:s0 /usr/bin/ovs-appctl
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/ovs-dpctl
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/ovs-dpctl-top
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/ovs-ofctl
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/bin/ovs-pki
-rwxr-xr-x. root root system_u:object_r:openvswitch_exec_t:s0 /usr/bin/ovs-vsctl


- In the other node, I didn't run restorecon but, instead, I reinstalled the openvswitch package to see if the files are correctly labeled:

[heat-admin@overcloud-compute-1 ~]$ ls -aZ /usr/bin/ovs-vsctl 
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/bin/ovs-vsctl

[heat-admin@overcloud-compute-1 ~]$ sudo rpm -Uvh openvswitch-2.6.1-10.git20161206.el7fdp.x86_64.rpm  --replacepkgs
Preparing...                          ################################# [100%]
Updating / installing...
   1:openvswitch-2.6.1-10.git20161206.################################# [100%]


[heat-admin@overcloud-compute-1 ~]$ ls -aZ /usr/bin/ovs-vsctl 
-rwxr-xr-x. root root system_u:object_r:openvswitch_exec_t:s0 /usr/bin/ovs-vsctl
[heat-admin@overcloud-compute-1 ~]$ 


- They are correctly labeled and no more denials in audit logs after that.


So, apparently, installing openvswitch in the overcloud image left the system mislabeled and we have to fix that. I'll try to reach out to someone else and will update this BZ accordingly.

Quoting [0], could you please try generating the overcloud image with --selinux-relabel option so that it gets correct SELinux labels? 

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1253623#c21

I'm closing this BZ since it looks like the image was generated only for testing and not using the right parameters as in [0]. Therefore, the SELinux labels were not correctly applied and ovs failed.

If you think that this requires to revisit, please re-open and I'll look into it.

Thanks,
Daniel

[0] https://github.com/redhat-openstack/infrared/blob/d38d396d20af238b1994e95ad5c48c8c80008b04/plugins/tripleo-undercloud/tasks/images/repos.yml#L77

It works with the change suggested [0]

[0] https://github.com/redhat-openstack/infrared/blob/d38d396d20af238b1994e95ad5c48c8c80008b04/plugins/tripleo-undercloud/tasks/images/repos.yml#L77