Bug 1496351

While we're at it I suggest to log also ovs-ctl output in something like /var/log/openvswitch/ovs-ctl.log

on the latest puddle, I saw in logs those error after fresh install:

[root@controller-1 ~]# tail -f  /var/log/openvswitch/ovsdb-server-nb.log
2017-10-01T10:09:06.369Z|04445|socket_util|ERR|6641:172.17.1.10: bind: Cannot assign requested address
2017-10-01T10:09:11.371Z|04450|socket_util|ERR|6641:172.17.1.10: bind: Cannot 
2017-10-01T10:09:18.876Z|04456|ovsdb_jsonrpc_server|ERR|Dropped 56 log messages in last 58 seconds (most recently, 3 seconds ago) due to excessive rate
2017-10-01T10:09:18.876Z|04457|ovsdb_jsonrpc_server|ERR|ptcp:6641:172.17.1.10: listen failed: Cannot assign requested address
2017-10-01T10:09:21.370Z|04458|socket_util|ERR|6641:172.17.1.10: bind: Cannot assign requested address

ADDED sos report

Created attachment 1332886 [details]
sosreport

^
|
 on the latest puddle(12   -p 2017-09-28.1)

^
|
 on the latest puddle(12   -p 2017-09-28.1)

numans explained that if we run 

ovs-appctl -t ovn-northd vlog/reopen


The logs start flowing again, so this looks like an ovs/ovn core issue.

Hi folks,

I'm a bit confused on this issue. I regularly set up small OVN test deployments with vagrant. Here is a link to a scenario I've used: https://github.com/putnopvut/vagrants/tree/master/ovn-3node

When I deploy OVN in this way, OVN logs are created as expected. The central node has north and southbound logs, as well as a northd log.

[vagrant@central ~]$ sudo ls -l /var/log/openvswitch
total 20
-rw-r-----. 1 root        root        586 Oct  3 14:36 ovn-northd.log
-rw-r-----. 1 openvswitch openvswitch 360 Oct  3 14:36 ovsdb-server.log
-rw-r-----. 1 root        root        363 Oct  3 14:36 ovsdb-server-nb.log
-rw-r-----. 1 root        root        364 Oct  3 14:36 ovsdb-server-sb.log
-rw-r-----. 1 openvswitch openvswitch 527 Oct  3 14:36 ovs-vswitchd.log

The controller nodes have an ovn-controller log file created.

[vagrant@compute1 ~]$ sudo ls -l /var/log/openvswitch/
total 12
-rw-r-----. 1 root        root        3056 Oct  3 14:37 ovn-controller.log
-rw-r-----. 1 openvswitch openvswitch  361 Oct  3 14:37 ovsdb-server.log
-rw-r-----. 1 openvswitch openvswitch 2884 Oct  3 14:37 ovs-vswitchd.log

To double-check, I tried this with both the upstream OVS master and the 2.7.2 release. In both cases, the logs appeared as expected. One difference I noticed between master and 2.7.2 was that in master, some log files are owned by root and others are owned by "openvswitch". In 2.7.2, all log files were owned by root.

Between the setup described in the first comment and my setup, the main differences I see are:
1) My vagrant setup pulls the source from upstream and builds the RPMs from source. On this issue, you're using fdProd RPMs.
2) I'm not using OpenStack. Rather, I'm just installing OVN and starting the services with systemd.

I doubt that (1) would make a difference but it was worth noting. With (2), that could make a difference since you could be starting the OVN services in some other way than I do, and it could be started by a different user than what I use.

When OpenStack starts OVN services, what user does it run as? Does it have permission to write to the /var/log/openvswitch directory? When you ran

ovs-appctl -t ovn-northd vlog/reopen

did you issue it as root? Or was it the same user that started the OVN services?

(In reply to Mark Michelson from comment #7)
> Hi folks,
> 
> I'm a bit confused on this issue. I regularly set up small OVN test
> deployments with vagrant. Here is a link to a scenario I've used:
> https://github.com/putnopvut/vagrants/tree/master/ovn-3node
> 
> When I deploy OVN in this way, OVN logs are created as expected. The central
> node has north and southbound logs, as well as a northd log.
> 
> [vagrant@central ~]$ sudo ls -l /var/log/openvswitch
> total 20
> -rw-r-----. 1 root        root        586 Oct  3 14:36 ovn-northd.log
> -rw-r-----. 1 openvswitch openvswitch 360 Oct  3 14:36 ovsdb-server.log
> -rw-r-----. 1 root        root        363 Oct  3 14:36 ovsdb-server-nb.log
> -rw-r-----. 1 root        root        364 Oct  3 14:36 ovsdb-server-sb.log
> -rw-r-----. 1 openvswitch openvswitch 527 Oct  3 14:36 ovs-vswitchd.log
> 
> The controller nodes have an ovn-controller log file created.
> 
> [vagrant@compute1 ~]$ sudo ls -l /var/log/openvswitch/
> total 12
> -rw-r-----. 1 root        root        3056 Oct  3 14:37 ovn-controller.log
> -rw-r-----. 1 openvswitch openvswitch  361 Oct  3 14:37 ovsdb-server.log
> -rw-r-----. 1 openvswitch openvswitch 2884 Oct  3 14:37 ovs-vswitchd.log
> 
> To double-check, I tried this with both the upstream OVS master and the
> 2.7.2 release. In both cases, the logs appeared as expected. One difference
> I noticed between master and 2.7.2 was that in master, some log files are
> owned by root and others are owned by "openvswitch". In 2.7.2, all log files
> were owned by root.
> 
> Between the setup described in the first comment and my setup, the main
> differences I see are:
> 1) My vagrant setup pulls the source from upstream and builds the RPMs from
> source. On this issue, you're using fdProd RPMs.
> 2) I'm not using OpenStack. Rather, I'm just installing OVN and starting the
> services with systemd.
> 
> I doubt that (1) would make a difference but it was worth noting. With (2),
> that could make a difference since you could be starting the OVN services in
> some other way than I do, and it could be started by a different user than
> what I use.
> 
> When OpenStack starts OVN services, what user does it run as? Does it have
> permission to write to the /var/log/openvswitch directory? When you ran
> 
> ovs-appctl -t ovn-northd vlog/reopen
> 
> did you issue it as root? Or was it the same user that started the OVN
> services?

Mark, the logs files created but they are empty.for some reason the system does not write to the log.

Can you provide an environment and set of steps to reproduce this? I have to admit I'm coming into this not having used OpenStack before, so any assistance towards reproducing the issue will be helpful.

I suspect that since this "works for me" the difference is all in the deployment of OVN and not in the source code of OVN.

Eran, can you provide @mark an environment where this has reproduced for him to log in and inspect what's exactly happening?

I looked into the /var/log/messages and /var/log/audit/audit.log files in the attached sosreport.
I could clearly see that ovn db servers are not able to open 6641/6642 ports because of selinix permission issues.

The BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1499134 should take care of it.

Still I expect the log files to be created.

@Eran - Once BZ 1499134 is fixed, can you please rerun again and verify this issue ?

(In reply to Numan Siddique from comment #12)
> I looked into the /var/log/messages and /var/log/audit/audit.log files in
> the attached sosreport.
> I could clearly see that ovn db servers are not able to open 6641/6642 ports
> because of selinix permission issues.
> 
> The BZ - https://bugzilla.redhat.com/show_bug.cgi?id=1499134 should take
> care of it.
> 
> Still I expect the log files to be created.
> 
> @Eran - Once BZ 1499134 is fixed, can you please rerun again and verify this
> issue ?

sure I will.

Numan, I rerun the test and the issue still exist.

@Eran - I am not seeing this issue with upstream tripleo deployment. I would like to get access to your setup if it's possible.

I submitted this patch series - https://patchwork.ozlabs.org/project/openvswitch/list/?series=12324  to address this issue.

The patches are merged upstream master, branch 2.8 and 2.7.

verified on osp12 z1 cat /etc/yum.repos.d/latest-installed 
12   -p 2018-01-24.2

rpm -qa |grep openvswitch
openvswitch-2.7.3-3.git20180112.el7fdp.x86_64
python-openvswitch-2.7.3-3.git20180112.el7fdp.noarch
openvswitch-ovn-common-2.7.3-3.git20180112.el7fdp.x86_64
openvswitch-ovn-central-2.7.3-3.git20180112.el7fdp.x86_64
openvswitch-ovn-host-2.7.3-3.git20180112.el7fdp.x86_64

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0248