First, some data harvested from the sosreport (I used sosreport-20171212-221628).

#CPU cores: 48
#internal ports in ovsdb (total over all ovs bridges): 4945
#ports added to the kernel datapath: 1868
#interfaces in root netns: 1871
#genetlink sockets opened by ovs-vswitchd: 64946

The actual limit of ports that could be added to ovs bridges (combined) on this system is 1868. This is surprisingly low; it's because each added port means ovs opens 35 netlink sockets on this machine.

From the ovs documentation (vswitchd/vswitch.xml):

<column name="other_config" key="n-handler-threads"
        type='{"type": "integer", "minInteger": 1}'>
  <p>
    Specifies the number of threads for software datapaths to use for
    handling new flows.  The default the number of online CPU cores minus
    the number of revalidators.
  </p>
  <p>
    This configuration is per datapath.  If you have more than one
    software datapath (e.g. some <code>system</code> bridges and some
    <code>netdev</code> bridges), then the total number of threads is
    <code>n-handler-threads</code> times the number of software
    datapaths.
  </p>
</column>

<column name="other_config" key="n-revalidator-threads"
        type='{"type": "integer", "minInteger": 1}'>
  <p>
    Specifies the number of threads for software datapaths to use for
    revalidating flows in the datapath.  Typically, there is a direct
    correlation between the number of revalidator threads, and the number
    of flows allowed in the datapath.  The default is the number of cpu
    cores divided by four plus one.  If <code>n-handler-threads</code> is
    set, the default changes to the number of cpu cores minus the number
    of handler threads.
  </p>
  <p>
    This configuration is per datapath.  If you have more than one
    software datapath (e.g. some <code>system</code> bridges and some
    <code>netdev</code> bridges), then the total number of threads is
    <code>n-handler-threads</code> times the number of software
    datapaths.
  </p>
</column>

Thus n-revalidator-threads is 13 and n-handler-threads is 35. But the documentation is in fact wrong: the 'handler' netlink sockets are opened *per port*. I have no idea whether it's a design thinko but it appears to me it is.

The ovs-vswitchd process has a limit of 65535 file descriptors. With the fds needed for other things, this leaves the number of possible ports to a mere 1868.

Now, the ovsdb contains 4945 internal ports. These are being added after ovs-vswitchd startup. When we reach the limit of 1868 ports, addition of further ports start to fail.

Possible workarounds:
- increasing the limit of file descriptors for ovs-vswitchd
- limiting n-handler-threads to 13 at most

(In reply to Jiri Benc from comment #4)
> - limiting n-handler-threads to 13 at most

ovs-vsctl set Open_vSwitch $(ovs-vsctl show | head -n 1) other_config:n-handler-threads=13

This is effective immediately (i.e., you should see number of opened file descriptors to decrease). But for me, it seems the number of opened netlink sockets per port is in fact 2 times n-handler-threads.

Looking at the code, this is designed this way:

static int
dpif_netlink_port_add__(struct dpif_netlink *dpif, struct netdev *netdev,
                        odp_port_t *port_nop)
{
    ...
    if (dpif->handlers) {
        socksp = vport_create_socksp(dpif, &error);
        ...
    }
    ...
    if (socksp) {
        error = vport_add_channels(dpif, *port_nop, socksp);
        ...
    }
    ...
}

static int 
vport_add_channels(struct dpif_netlink *dpif, odp_port_t port_no,
                   struct nl_sock **socksp)
{       
    ...
    uint32_t port_idx = odp_to_u32(port_no);
    ...
    ... realloc dpif->handler[i]->channels if needed
    ...
    for (i = 0; i < dpif->n_handlers; i++) {
        ...
        dpif->handlers[i].channels[port_idx].sock = socksp[i];
        ...
    }
    ...
}

I don't see the motivation behind that. Changing it will be a substantial effort, though.

We are currently trying this workaround to increase the number of file descriptors for ovs-vswitchd --- the following is from my lab. I'm getting back as soon as I have the customer's answer.

++++++++++++++++++++++++++++

Can we try to increase the number of file descriptors for ovs-vswitchd significantly:
~~~
mkdir /etc/systemd/system/ovs-vswitchd.service.d/
echo -e "[Service]\nLimitNOFILE=262144" > /etc/systemd/system/ovs-vswitchd.service.d/limit_no_file.conf
systemctl daemon-reload
systemctl restart openvswitch
~~~

After this change, `systemctl status ovs-vswitchd` should show:
~~~
systemctl status ovs-vswitchd
(...)
  Drop-In: /etc/systemd/system/ovs-vswitchd.service.d
           └─limit_no_file.conf
(...)
~~~

Finally, verify the max open files for the ovs-vswitchd process:
~~~
[root@undercloud-8 ~]# cat /proc/`pidof ovs-vswitchd`/limits | grep 'Max open'
Max open files            262144               262144               files  
~~~

I kept thinking about this over the weekend. Regardless of the ovs bug (or misdesign?) I found on Friday, we should clarify one thing: why there are so many internal ports in ovsdb? Surely there could have never been a point in time where all of those ports were operating together (because of, obviously, not enough file descriptors).

Is it because in the past, there were 1868 VMs running (or, possibly, 934 or another fraction of that number depending on how many ports are needed for a single VM) and OpenStack tried to add one more VM? And when it failed, it retried, etc. ad infinitum? Or what is the cause?

(In reply to Jiri Benc from comment #13)
> I kept thinking about this over the weekend. Regardless of the ovs bug (or
> misdesign?) I found on Friday, we should clarify one thing: why there are so
> many internal ports in ovsdb? Surely there could have never been a point in
> time where all of those ports were operating together (because of,
> obviously, not enough file descriptors).
> 
> Is it because in the past, there were 1868 VMs running (or, possibly, 934 or
> another fraction of that number depending on how many ports are needed for a
> single VM) and OpenStack tried to add one more VM? And when it failed, it
> retried, etc. ad infinitum? Or what is the cause?

That's a great question and we definitely need to find the answer for it. Thanks for your analysis Jiri, I learnt about how file descriptors are consumed by OVS and how to tweak it.
Regarding your question and without having looked into the sosreports yet, the other day we hit some bug that whenever a VM was created, for some unknown reason, two neutron ports were created (even though it had only one NIC). Apparently, this bug is rare and was observed in the past but we weren't able to collect logs. I wonder whether this environment could be experiencing the same issue... As I'm a bit late to the party, we could try to sync tomorrow and I'll check neutron/nova logs to see if I can find a lead.

I've been trying to categorize the ports in the system:

* Total ports are 4965:
$ cat ovs-vsctl_-t_5_show  | grep Port | wc -l
4965

* qr ports are 260:
$ cat ovs-vsctl_-t_5_show  | grep Port | grep "qr-" | wc -l
260

* qg ports are 260:
$ cat ovs-vsctl_-t_5_show  | grep Port | grep "qg-" | wc -l
263

* tap ports (DHCP) are 3312:
$ cat ovs-vsctl_-t_5_show  | grep Port | grep "tap" | wc -l
3312

* ha ports are 1107:
$ cat ovs-vsctl_-t_5_show  | grep Port | grep "ha-" | wc -l
1107

* vxlan ports are 15:
$ cat ovs-vsctl_-t_5_show  | grep Port | grep "vxlan" | wc -l
15


The most ports are tap ports (DHCP) but in the sosreport I can see fewer namespaces as per this command:
$ cat ip_netns | grep qdhcp | wc -l
192

$ cat ip_netns | grep qrouter | wc -l
11

In neutron logs I can see lots of issues connecting to rabbit.
Also, DHCP agent log is full of errors (ovsdb timeouts, cannot find device tapXXX, etc.).

Questions on my side:

- How many networks, tenants and ha routers do they have?
- How's dhcp configured? Is it active/active?

Thanks,
Daniel

Sharing privately customer reply

Hi,

Let me please clarify the instructions I'd suggest to fix the current issue is:

1. Run ovs-cleanup the script this way:

  1.1 Create cleanup.conf file which looks like this inside /etc/neutron/conf.d/neutron-ovs-cleanup:
    [DEFAULT]
    ovs_vsctl_timeout=1800

  1.2 Execute the ovs cleanup script with debug enabled:
    /usr/bin/neutron-ovs-cleanup --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-ovs-cleanup --log-file /var/log/neutron/ovs-cleanup.log --debug

  At this point we expect that unused ovs devices are gone and unused namespaces are empty.

2. Run netns-cleanup with debug and *without* --force:

  /usr/bin/neutron-netns-cleanup --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-netns-cleanup --log-file /var/log/neutron/netns-cleanup.log --debug

  At this point we expect all the empty namespaces to be deleted.

3. Enable debug in /etc/neutron/neutron.conf and restart neutron services.


Also fix the logrotate issue because with debug enabled, log files will be way heavier so it's good that we rotate them.

Please, we note that this is providing a sort of workaround because of something that failed before and made those so many ports to be left behind. Also, the ovs-cleanup script didn't work properly which may have helped as in logs we can see that it was run but failed. For us to be able to identify the root cause if it happens again, we suggest to leave debug enabled and monitor the number of ports for a certain amount of time.

Please, let us know your thoughts and/or questions.

The native ovsdb implementation is sometimes slower when deleting lots of ports. Another thing to try adding to Daniel's comment #19 /etc/neutron/conf.d/neutron-ovs-cleanup file is

[OVS]
ovsdb_interface=vsctl

and see if it runs considerably faster. I'm currently investigating the performance differences between the two. It would be good to see if the vsctl interface is faster in a real-world scenario. If so, I need to invest time in optimizing the the native interface in this case.

Sent an RFC to the ovs-dev mailing list:

https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343508.html

Created attachment 1452875 [details]
patch for the kernel data path

Created attachment 1454715 [details]
beta patch for the kernel data path

Maybe this can be approached from the opposite end? How is the fairness ensured with the current solution? There's a finite number of threads reading from a much higher number of sockets. How is the socket to be processed next selected in the user space? Is the fairness really guaranteed? How do the ovs user space locks affect the processing?

If there's a code to ensure fairness, it can probably be replicated on the kernel side. After all, we do have a serializing point currently, as we go from a high number of sockets to a low number of threads. We're just moving this point a little bit down to the kernel.

(In reply to Jiri Benc from comment #70)
> If there's a code to ensure fairness, it can probably be replicated on the
> kernel side.

My understanding is that ovs-switchd opens a huge number of sockets and polls them, and this results in a (rather random, but somewhat fair) round-robin fashion. The proposals outlined above are already an attempt at replicating this.

RFC upstream patch at:

    https://patchwork.ozlabs.org/patch/939390/

Created attachment 1477839 [details]
tool to change the datapath round robin algorithm

Created attachment 1477840 [details]
beta patch for the kernel data path

Created attachment 1483331 [details]
patch for userspace vswitchd

I wrote a tool[1] which calculates the time that a packet sent to a veth outside the bridge needs to do all the travel through the enslaved peer, and then to the bridge passing by the slow path.

Before the patch:

root@wsfd-netdev87:~# nproc
56

root@wsfd-netdev87:~# ovs-vsctl show |grep -c Port
2223

root@wsfd-netdev87:~# lsof -p $(pidof ovs-vswitchd) |grep -c GENERIC
91187

Idle system

root@wsfd-netdev87:~# for i in {1..22}; do echo -n "Port$i: "; weed
veth-ext$i ovs_upc_br -i10u -c 10k |awk -F/ '/^eed/{print$4}'; done
Port1: 1020
Port2: 1011
Port3: 1010
Port4: 1035
Port5: 1190
Port6: 1019
Port7: 1027
Port8: 967
Port9: 996
Port10: 986
Port11: 1090
Port12: 1004
Port13: 1070
Port14: 978
Port15: 971
Port16: 1023
Port17: 1003
Port18: 1007
Port19: 996
Port20: 1023
Port21: 961
Port22: 1018

System under heavy load

root@wsfd-netdev87:~# upc_test.sh start
root@wsfd-netdev87:~# weed veth-ext21 ovs_upc_br -c100 -v
eed: 7938 ns
eed: 8183 ns
eed: 6660 ns
eed: 7585 ns
eed: 7327 ns
100 packets transmitted, 5 received, 95% packet loss
eed min/avg/max = 6660/7538/8183 ns


With the patch:

root@wsfd-netdev87:~# nproc
56

root@wsfd-netdev87:~# ovs-vsctl show |grep -c Port
2223

root@wsfd-netdev87:~# lsof -p $(pidof ovs-vswitchd) |grep -c GENERIC
2227

Idle system

root@wsfd-netdev87:~# for i in {1..22}; do echo -n "Port$i: "; weed
veth-ext$i ovs_upc_br -i10u -c 10k |awk -F/ '/^eed/{print$4}'; done
Port1: 917
Port2: 923
Port3: 844
Port4: 908
Port5: 908
Port6: 859
Port7: 884
Port8: 935
Port9: 856
Port10: 848
Port11: 850
Port12: 874
Port13: 937
Port14: 877
Port15: 907
Port16: 910
Port17: 910
Port18: 887
Port19: 847
Port20: 918
Port21: 809
Port22: 823

System under heavy load

root@wsfd-netdev87:~# upc_test.sh start
root@wsfd-netdev87:~# weed veth-ext21 ovs_upc_br -c100 -v
eed: 4213 ns
eed: 3851 ns
eed: 3585 ns
eed: 3155 ns
eed: 4310 ns
eed: 4043 ns
eed: 3662 ns
eed: 4088 ns
eed: 3227 ns
eed: 4697 ns
eed: 4077 ns
eed: 3844 ns
eed: 4349 ns
eed: 3460 ns
eed: 4313 ns
eed: 3505 ns
eed: 4216 ns
eed: 4461 ns
eed: 3212 ns
eed: 3037 ns
eed: 4249 ns
eed: 3336 ns
eed: 2923 ns
eed: 4559 ns
eed: 4665 ns
eed: 2920 ns
eed: 3744 ns
eed: 3390 ns
eed: 4635 ns
eed: 3965 ns
eed: 3568 ns
eed: 3708 ns
eed: 3525 ns
eed: 4154 ns
eed: 3090 ns
eed: 3696 ns
eed: 3329 ns
eed: 3844 ns
100 packets transmitted, 38 received, 62% packet loss
eed min/avg/max = 2920/3805/4697 ns

Regards,
Matteo

[1] https://github.com/teknoraver/network-tools

(In reply to Matteo Croce from comment #79)
> Created attachment 1483331 [details]
> patch for userspace vswitchd
> 
> I wrote a tool[1] which calculates the time that a packet sent to a veth
> outside the bridge needs to do all the travel through the enslaved peer, and
> then to the bridge passing by the slow path.

How does this measure per-vport fairness, which was the actual concern raised in [1]?

[1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343575.html

Hi Stefano,

I made a better test to mesure fairness. I setup a bridge with 22 ports, and inject traffic in all but one port.

Then I start the latency measure on the idle port, to see if the packets are lost or not.

This is the test with stock OVS:

# weed veth-ext1 ovs_upc_br -i1m -c100k
100000 packets transmitted, 5172 received, 94% packet loss
eed min/avg/max/mdev = 1.1/6.1/18.3/2.2 us

and this with the userspace patch:

# weed veth-ext1 ovs_upc_br -i1m -c100k
100000 packets transmitted, 95032 received, 4% packet loss
eed min/avg/max/mdev = 1.0/2.1/344.5/1.2 us

Patch submitted upstream

https://mail.openvswitch.org/pipermail/ovs-dev/2018-September/352251.html

(In reply to Matteo Croce from comment #81)
> Hi Stefano,
> 
> I made a better test to mesure fairness. I setup a bridge with 22 ports, and
> inject traffic in all but one port.
> 
> Then I start the latency measure on the idle port, to see if the packets are
> lost or not.
> 
> This is the test with stock OVS:
> 
> # weed veth-ext1 ovs_upc_br -i1m -c100k
> 100000 packets transmitted, 5172 received, 94% packet loss
> eed min/avg/max/mdev = 1.1/6.1/18.3/2.2 us
> 
> and this with the userspace patch:
> 
> # weed veth-ext1 ovs_upc_br -i1m -c100k
> 100000 packets transmitted, 95032 received, 4% packet loss
> eed min/avg/max/mdev = 1.0/2.1/344.5/1.2 us

As I mentioned on IRC: you are keeping 21 ovs-vswitchd threads busy out of 56 CPU threads (hence, 56 possible ovs-vswitchd threads) of your test host. I think you should rather saturate ovs-vswitchd threads to have valid results on this test. Otherwise, ovs-vswitchd can just handle the "control" traffic on a thread which isn't loaded at all.

You could e.g. limit the number of CPU threads ovs-vswitchd can run on with taskset(1).

But let's see, maybe it's good enough for upstream.

(In reply to Matteo Croce from comment #82)
> Patch submitted upstream
> 
> https://mail.openvswitch.org/pipermail/ovs-dev/2018-September/352251.html

I'm not sure where you #define EPOLLEXCLUSIVE, but it looks like you simply don't use that flag if it's not available (say, Linux kernel versions < 4.5, RHEL versions < 7.3) without keeping per-thread sockets, which, according to commit message, would lead to duplicate events. But perhaps this is also acceptable for upstream.

Hi Stefano,

I don't #define EPOLLEXCLUSIVE myself it's defined in /usr/include/sys/epoll.h.

Right, with older kernels than 4.5 or RHEL 7.3, I just don't use the flag.
This of course leads to duplicate events, but even without the patch, many events are duplicated, probably for the way the hashing of the port id in the kernel works. This can be observed by adding a printf in dpif_netlink_recv__().

This seems not to be an issue, because even if multiple threads are waken, only one can read the packet, so there are no duplicate packets, this causes just unnecessary wake up of the handler threads.

With the patch and EPOLLEXCLUSIVE instead, exactly only one thread is waken up, leading to lower latency.

Cheers,

Patch merged upstream:

https://mail.openvswitch.org/pipermail/ovs-dev/2018-September/352402.html

Reproduce with openvswitch-2.9.0-56.el7fdp.x86_64


[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  1  10  11  12  13  14  15  16  17  2  3  4  5  6  7  8  9
[root@dell-per730-04 ~]# ovs-vsctl add-br ovsbr0
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0    101  105  109  112  116  12   123  127  130  134  138  141  145  149  152  156  16  2   23  27  30  34  38  41  45  49  52  56  6   63  67  70  74  78  81  85  89  92  96
1    102  106  11   113  117  120  124  128  131  135  139  142  146  15   153  157  17  20  24  28  31  35  39  42  46  5   53  57  60  64  68  71  75  79  82  86  9   93  97
10   103  107  110  114  118  121  125  129  132  136  14   143  147  150  154  158  18  21  25  29  32  36  4   43  47  50  54  58  61  65  69  72  76  8   83  87  90  94  98
100  104  108  111  115  119  122  126  13   133  137  140  144  148  151  155  159  19  22  26  3   33  37  40  44  48  51  55  59  62  66  7   73  77  80  84  88  91  95  99
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p0 -- set interface p0 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0    102  107  111  116  120  125  13   134  139  143  148  152  157  161  166  170  175  18   184  189  193  22  27  31  36  40  45  5   54  59  63  68  72  77  81  86  90  95
1    103  108  112  117  121  126  130  135  14   144  149  153  158  162  167  171  176  180  185  19   194  23  28  32  37  41  46  50  55  6   64  69  73  78  82  87  91  96
10   104  109  113  118  122  127  131  136  140  145  15   154  159  163  168  172  177  181  186  190  2    24  29  33  38  42  47  51  56  60  65  7   74  79  83  88  92  97
100  105  11   114  119  123  128  132  137  141  146  150  155  16   164  169  173  178  182  187  191  20   25  3   34  39  43  48  52  57  61  66  70  75  8   84  89  93  98
101  106  110  115  12   124  129  133  138  142  147  151  156  160  165  17   174  179  183  188  192  21   26  30  35  4   44  49  53  58  62  67  71  76  80  85  9   94  99
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p1 -- set interface p1 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0    103  109  114  12   125  130  136  141  147  152  158  163  169  174  18   185  190  196  200  206  211  217  222  228  27  32  38  43  49  54  6   65  70  76  81  87  92  98
1    104  11   115  120  126  131  137  142  148  153  159  164  17   175  180  186  191  197  201  207  212  218  223  229  28  33  39  44  5   55  60  66  71  77  82  88  93  99
10   105  110  116  121  127  132  138  143  149  154  16   165  170  176  181  187  192  198  202  208  213  219  224  23   29  34  4   45  50  56  61  67  72  78  83  89  94
100  106  111  117  122  128  133  139  144  15   155  160  166  171  177  182  188  193  199  203  209  214  22   225  24   3   35  40  46  51  57  62  68  73  79  84  9   95
101  107  112  118  123  129  134  14   145  150  156  161  167  172  178  183  189  194  2    204  21   215  220  226  25   30  36  41  47  52  58  63  69  74  8   85  90  96
102  108  113  119  124  13   135  140  146  151  157  162  168  173  179  184  19   195  20   205  210  216  221  227  26   31  37  42  48  53  59  64  7   75  80  86  91  97
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p2 -- set interface p2 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0    103  109  114  12   125  130  136  141  147  152  158  163  169  174  18   185  190  196  200  206  211  217  222  228  233  239  244  25   255  260  28  33  39  44  5   55  60  66  71  77  82  88  93  99
1    104  11   115  120  126  131  137  142  148  153  159  164  17   175  180  186  191  197  201  207  212  218  223  229  234  24   245  250  256  261  29  34  4   45  50  56  61  67  72  78  83  89  94
10   105  110  116  121  127  132  138  143  149  154  16   165  170  176  181  187  192  198  202  208  213  219  224  23   235  240  246  251  257  262  3   35  40  46  51  57  62  68  73  79  84  9   95
100  106  111  117  122  128  133  139  144  15   155  160  166  171  177  182  188  193  199  203  209  214  22   225  230  236  241  247  252  258  263  30  36  41  47  52  58  63  69  74  8   85  90  96
101  107  112  118  123  129  134  14   145  150  156  161  167  172  178  183  189  194  2    204  21   215  220  226  231  237  242  248  253  259  264  31  37  42  48  53  59  64  7   75  80  86  91  97
102  108  113  119  124  13   135  140  146  151  157  162  168  173  179  184  19   195  20   205  210  216  221  227  232  238  243  249  254  26   27   32  38  43  49  54  6   65  70  76  81  87  92  98
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p3 -- set interface p3 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0    104  110  117  123  13   136  142  149  155  161  168  174  180  187  193  2    205  211  218  224  230  237  243  25   256  262  269  275  281  288  294  30  37  43  5   56  62  69  75  81  88  94
1    105  111  118  124  130  137  143  15   156  162  169  175  181  188  194  20   206  212  219  225  231  238  244  250  257  263  27   276  282  289  295  31  38  44  50  57  63  7   76  82  89  95
10   106  112  119  125  131  138  144  150  157  163  17   176  182  189  195  200  207  213  22   226  232  239  245  251  258  264  270  277  283  29   296  32  39  45  51  58  64  70  77  83  9   96
100  107  113  12   126  132  139  145  151  158  164  170  177  183  19   196  201  208  214  220  227  233  24   246  252  259  265  271  278  284  290  297  33  4   46  52  59  65  71  78  84  90  97
101  108  114  120  127  133  14   146  152  159  165  171  178  184  190  197  202  209  215  221  228  234  240  247  253  26   266  272  279  285  291  298  34  40  47  53  6   66  72  79  85  91  98
102  109  115  121  128  134  140  147  153  16   166  172  179  185  191  198  203  21   216  222  229  235  241  248  254  260  267  273  28   286  292  299  35  41  48  54  60  67  73  8   86  92  99
103  11   116  122  129  135  141  148  154  160  167  173  18   186  192  199  204  210  217  223  23   236  242  249  255  261  268  274  280  287  293  3    36  42  49  55  61  68  74  80  87  93
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p4 -- set interface p4 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0    105  112  12   127  134  141  149  156  163  170  178  185  192  2    206  213  220  228  235  242  25   257  264  271  279  286  293  30   307  314  321  329  35  42  5   57  64  71  79  86  93
1    106  113  120  128  135  142  15   157  164  171  179  186  193  20   207  214  221  229  236  243  250  258  265  272  28   287  294  300  308  315  322  33   36  43  50  58  65  72  8   87  94
10   107  114  121  129  136  143  150  158  165  172  18   187  194  200  208  215  222  23   237  244  251  259  266  273  280  288  295  301  309  316  323  330  37  44  51  59  66  73  80  88  95
100  108  115  122  13   137  144  151  159  166  173  180  188  195  201  209  216  223  230  238  245  252  26   267  274  281  289  296  302  31   317  324  331  38  45  52  6   67  74  81  89  96
101  109  116  123  130  138  145  152  16   167  174  181  189  196  202  21   217  224  231  239  246  253  260  268  275  282  29   297  303  310  318  325  332  39  46  53  60  68  75  82  9   97
102  11   117  124  131  139  146  153  160  168  175  182  19   197  203  210  218  225  232  24   247  254  261  269  276  283  290  298  304  311  319  326  333  4   47  54  61  69  76  83  90  98
103  110  118  125  132  14   147  154  161  169  176  183  190  198  204  211  219  226  233  240  248  255  262  27   277  284  291  299  305  312  32   327  334  40  48  55  62  7   77  84  91  99
104  111  119  126  133  140  148  155  162  17   177  184  191  199  205  212  22   227  234  241  249  256  263  270  278  285  292  3    306  313  320  328  34   41  49  56  63  70  78  85  92
[root@dell-per730-04 ~]# 



Verify with openvswitch-2.9.0-70.el7fdp.x86_64

[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  1  10  11  12  13  14  15  16  17  2  3  4  5  6  7  8  9
[root@dell-per730-04 ~]# ovs-vsctl add-br ovsbr0
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  65  67  69  70  72  74  76  78  8   81  83  85  87  89  90
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  64  66  68  7   71  73  75  77  79  80  82  84  86  88  9   91
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p0 -- set interface p0 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  65  67  69  70  72  74  76  78  8   81  83  85  87  89  90  92
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  64  66  68  7   71  73  75  77  79  80  82  84  86  88  9   91
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p1 -- set interface p1 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  65  67  69  70  72  74  76  78  8   81  83  85  87  89  90  92
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  64  66  68  7   71  73  75  77  79  80  82  84  86  88  9   91  93
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p2 -- set interface p2 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  65  67  69  70  72  74  76  78  8   81  83  85  87  89  90  92  94
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  64  66  68  7   71  73  75  77  79  80  82  84  86  88  9   91  93
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p3 -- set interface p3 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  65  67  69  70  72  74  76  78  8   81  83  85  87  89  90  92  94
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  64  66  68  7   71  73  75  77  79  80  82  84  86  88  9   91  93  95
[root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p4 -- set interface p4 type=internal
[root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  65  67  69  70  72  74  76  78  8   81  83  85  87  89  90  92  94  96
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  64  66  68  7   71  73  75  77  79  80  82  84  86  88  9   91  93  95

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3500