Do we know why the port is needed?

We don't know why openvswitch attempts to connect to a port 3310. That port is weird. I suggest to wait with allowing this port in our policy before proceeding, we could create a hole instead of fixing bug.

Ryan, what's the rationale to allow that port? Do we know why it shows up in OSP?

@Terry - Even though SELinux was already modified to open the port, I think it's our responsibility to investigate why we need the port opened. Let's make sure we understand why.

type=AVC msg=audit(1475756683.078:8897): avc:  denied  { name_connect } for pid=26463 comm="ovs-vswitchd" dest=3310 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:clamd_port_t:s0 tclass=tcp_socket

Clamd is an anti virus daemon that listens for incoming connections and scans desired files/dirs for threats.

This rule allows for ovs to connect to clamd_port_t so clamd can receive requests.

(In reply to Ryan Hallisey from comment #7)
> type=AVC msg=audit(1475756683.078:8897): avc:  denied  { name_connect } for
> pid=26463 comm="ovs-vswitchd" dest=3310
> scontext=system_u:system_r:openvswitch_t:s0
> tcontext=system_u:object_r:clamd_port_t:s0 tclass=tcp_socket
> 
> Clamd is an anti virus daemon that listens for incoming connections and
> scans desired files/dirs for threats.
> 
> This rule allows for ovs to connect to clamd_port_t so clamd can receive
> requests.

Yeah, it doesn't make sense to me that ovs would try to connect to anti virus. openvswitch typically connects to controller. We need to investigate first whether this is a test only issue, I doubt this happens in real deployments.

Ryan: That sounds very odd to me. Why would the ovs-vswitchd process need to communicate with clam-av? I have not heard anything that would convince me that this should be in the default policy. Who is configuring ovs-vswitchd to make outgoing connections to clam-av? The only way I can think of is having it connect to it as a controller, which seems crazy unless clam-av has a built-in openflow controller that I'm not aware of.

Ok if this is odd behaviour then we need to get to the root of why ovs is trying to connect to clam-av. Could this be a misconfiguration?

I've been running the job several times and here are the results (each line is from a different build):


ovs-vswitchd: ovs|11280|stream_tcp|ERR|tcp:127.0.0.1:3310: connect: Permission denied

ovs-vswitchd: ovs|10654|stream_tcp|ERR|tcp:127.0.0.1:6543: connect: Permission denied

ovs-vswitchd: ovs|10977|stream_tcp|ERR|tcp:127.0.0.1:6792: connect: Permission denied

ovs-vswitchd: ovs|12065|stream_tcp|ERR|tcp:127.0.0.1:9280: connect: Permission denied

ovs-vswitchd: ovs|14714|stream_tcp|ERR|tcp:127.0.0.1:10081: connect: Permission denied

ovs-vswitchd: ovs|11698|stream_tcp|ERR|tcp:127.0.0.1:5004: connect: Permission denied

OVSAgentTestBase sets a random of_listen_port in the config, and some tests call set_controllers() using this port. This causes ovs-vswitchd to attempt to make an outbound connection on this port, and thus SELinux being very cross with it.

Since this is a testing only issue, we should be able to remove the blocker flag. We absolutely need to remove any ports allowed in openstack-selinux from this change.

As far as solutions go, we could probably modify the tests to tell SELinux to allow the random port returned (or just allow ovs-vswitchd to connect anywhere in the test environment only, though I prefer a more targeted approach).

The permission error leads to a loop with the following log messages:

2016-10-10 16:07:03.793 27938 DEBUG ryu.base.app_manager [-] EVENT OVSNeutronAgentRyuApp->ofctl_service GetDatapathRequest send_event /usr/lib/python2.7/site-packages/ryu/base/app_manager.py:314
2016-10-10 16:07:03.794 27938 DEBUG OfctlService [-] dpid 165586572766272 -> datapath None _handle_get_datapath /usr/lib/python2.7/site-packages/ryu/app/ofctl/service.py:106

A timeout of some kind should probably be added upstream to prevent infinite looping on error.

I took the liberty to remove the blocker flag in light of Terry's comment 15: "Since this is a testing only issue..."

sounds good. Thanks everyone :)

Thanks Terry and Assaf for the investigation!

We have a workaround, moving to 10.z regardless.

Ryan: I have some questions about the os-ovs policy. It looks like from another bug, it was already added that OVS should be able to connect outbound to any unreserved_port_t

[terry@centos7 yum.repos.d]$ sudo sepolicy network -t unreserved_port_t (https://github.com/redhat-openstack/openstack-selinux/blob/37338757f3f929d2cce9e60d6ac464297bc7c5f2/os-ovs.te#L32)

With unreserved_port_t being:
unreserved_port_t: tcp: 1024-32767,61001-65535
unreserved_port_t: udp: 1024-32767,61001-65535

I have several questions about this:
1) Is it good SELinux policy to allow, by default, outbound connections anywhere just because it is technically possible to configure OVS that way? I thought the idea was generally to deny everything, then specifically allow only exactly what was needed?

2) If the intention really is "allow ovs to connect anywhere", unreserved_port_t leaves out reserved_port_t (1-511), hi_reserved_port_t(511-1023), and ephemeral_port_t(32768-61000). Should those be added?

3) Despite the listed port (3310) being in the range unreserved_port_t covers, it is still denied since clamd_port_t is defined. I've tested this with some other defined ports, and it appears that any defined port will cause the SELinux failure, so the above doesn't seem to actually accomplish what it set out to do. Is there actually a way to match all ports, defined or not, and allow them?

The type unreserved_port_t is given to ports that are not reserved for a particular service. So this covers all the ports that aren't specified in `sudo semanage port -l` within 1024-32767 and 61001-65535. Since OVS can connect to any port, we'll allow it to connect to those unreserved ports.

The intention is to allow things that are requested and make sense. Since there hasn't been a request for anything with hi_reserved_port_t or ephermeral_port_t yet, I'll wait on adding that policy.

This case had the port labeled (clamd_port_t).  If it is labeled and the connection to that port makes sense, then we can allow it.

> The type unreserved_port_t is given to ports that are not reserved for a particular service. So this covers all the ports that aren't specified in `sudo semanage port -l` within 1024-32767 and 61001-65535. Since OVS can connect to any port, we'll allow it to connect to those unreserved ports.

I think both of these bugs are about binding to an address. I mistakenly mentioned outbound connections earlier. I'm not an selinux expert, so I'm just trying to understand how things should be done and still have questions:

1) For the previous NSX bug, it seems like we would preferably want a very specific port allowed. Port 6640 is the standard IANA assigned port for OVSDB communication--it seems like it makes sense to specifically add it instead of unreserved_port_t. Much like how httpd can be configured to bind to any address but we have rules limiting tcp_socket name_bind http_port_t.

2) For this bug, the port used is completely random since it is used for testing purposes and multiple tests can run simultaneously and need to run from different ports. Just allowing clamd_port_t isn't beneficial, so the new rule needs to be removed.

3) If we are going to try to allow the functional tests to run with selinux in enforcing mode, then **for test environments only** we have to come up with a way to either: a) limit the "random" ports to ports that match unreserved_port_t/aren't defined elsewhere in selinux (if we keep unreserved_port_t), b) allow ovs-vswitchd to bind to *any* port, not just unreserved_port_t, or c) somehow specifically enable each random port before using it during the test (seems like it would be slow and we would still conflict with existing defined ports). I, not being an selinux expert, don't actually know how to do any of those things. Any ideas?

Sure!  Questions welcome :)

If you need a port labeled that is currently set as unreserved_port_t then that would go into base selinux policy.  We shouldn't just carry it for openstack-selinux.  Can you file a bug with selinux for that?

Since this is testing and the port is random then it doesn't make sense to add the rule. I agree. I removed it a few builds ago.

This rule will allow ovs to connect to any port.
  corenet_tcp_connect_all_ports(openvswitch_t)

To build policy for testing:

[rhallisey@localhost ~]$ cat os-testing.te
policy_module(os-testing,0.1)

gen_require(`
        type openvswitch_t;
')

# For testing OVS only
corenet_tcp_connect_all_ports(openvswitch_t)
[rhallisey@localhost ~]$ make -f /usr/share/selinux/devel/Makefile
[rhallisey@localhost ~]$ sudo semodule -i os-testing.pp