Bug 1526306
| Summary: | ovs-vswitchd service hangs with Error too many open files | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Nikhil Shetty <nshetty> | ||||||||||||||
| Component: | openvswitch | Assignee: | Matteo Croce <mcroce> | ||||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | qding | ||||||||||||||
| Severity: | medium | Docs Contact: | |||||||||||||||
| Priority: | high | ||||||||||||||||
| Version: | 7.4 | CC: | akaris, amuller, apevec, atelang, atragler, bjarolim, chrisw, ctrautma, dalvarez, haili, jbenc, jlibosva, jraju, kzhang, lmanasko, lmiksik, mcroce, nshetty, ovs-team, pbarta, pmorey, pvauter, qding, ragiman, rcernin, rhos-maint, rkhan, rlondhe, sbrivio, srevivo, sukulkar, tkajinam, tredaelli, twilson | ||||||||||||||
| Target Milestone: | pre-dev-freeze | ||||||||||||||||
| Target Release: | --- | ||||||||||||||||
| Hardware: | Unspecified | ||||||||||||||||
| OS: | Unspecified | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Fixed In Version: | openvswitch-2.9.0-66 | Doc Type: | Bug Fix | ||||||||||||||
| Doc Text: |
Previously, when the ovs-vswitchd service restarted, an error displayed with many open files. With this update, the number of sockets opened by ovs-vswitchd is decreased. As a result, the described problem no longer occurs.
|
Story Points: | --- | ||||||||||||||
| Clone Of: | |||||||||||||||||
| : | 1532280 1634015 (view as bug list) | Environment: | |||||||||||||||
| Last Closed: | 2018-11-05 14:59:03 UTC | Type: | Bug | ||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
| Embargoed: | |||||||||||||||||
| Bug Depends On: | |||||||||||||||||
| Bug Blocks: | 1532280, 1543894, 1543906 | ||||||||||||||||
| Attachments: |
|
||||||||||||||||
|
Description
Nikhil Shetty
2017-12-15 07:58:44 UTC
First, some data harvested from the sosreport (I used sosreport-20171212-221628).
#CPU cores: 48
#internal ports in ovsdb (total over all ovs bridges): 4945
#ports added to the kernel datapath: 1868
#interfaces in root netns: 1871
#genetlink sockets opened by ovs-vswitchd: 64946
The actual limit of ports that could be added to ovs bridges (combined) on this system is 1868. This is surprisingly low; it's because each added port means ovs opens 35 netlink sockets on this machine.
From the ovs documentation (vswitchd/vswitch.xml):
<column name="other_config" key="n-handler-threads"
type='{"type": "integer", "minInteger": 1}'>
<p>
Specifies the number of threads for software datapaths to use for
handling new flows. The default the number of online CPU cores minus
the number of revalidators.
</p>
<p>
This configuration is per datapath. If you have more than one
software datapath (e.g. some <code>system</code> bridges and some
<code>netdev</code> bridges), then the total number of threads is
<code>n-handler-threads</code> times the number of software
datapaths.
</p>
</column>
<column name="other_config" key="n-revalidator-threads"
type='{"type": "integer", "minInteger": 1}'>
<p>
Specifies the number of threads for software datapaths to use for
revalidating flows in the datapath. Typically, there is a direct
correlation between the number of revalidator threads, and the number
of flows allowed in the datapath. The default is the number of cpu
cores divided by four plus one. If <code>n-handler-threads</code> is
set, the default changes to the number of cpu cores minus the number
of handler threads.
</p>
<p>
This configuration is per datapath. If you have more than one
software datapath (e.g. some <code>system</code> bridges and some
<code>netdev</code> bridges), then the total number of threads is
<code>n-handler-threads</code> times the number of software
datapaths.
</p>
</column>
Thus n-revalidator-threads is 13 and n-handler-threads is 35. But the documentation is in fact wrong: the 'handler' netlink sockets are opened *per port*. I have no idea whether it's a design thinko but it appears to me it is.
The ovs-vswitchd process has a limit of 65535 file descriptors. With the fds needed for other things, this leaves the number of possible ports to a mere 1868.
Now, the ovsdb contains 4945 internal ports. These are being added after ovs-vswitchd startup. When we reach the limit of 1868 ports, addition of further ports start to fail.
Possible workarounds: - increasing the limit of file descriptors for ovs-vswitchd - limiting n-handler-threads to 13 at most (In reply to Jiri Benc from comment #4) > - limiting n-handler-threads to 13 at most ovs-vsctl set Open_vSwitch $(ovs-vsctl show | head -n 1) other_config:n-handler-threads=13 This is effective immediately (i.e., you should see number of opened file descriptors to decrease). But for me, it seems the number of opened netlink sockets per port is in fact 2 times n-handler-threads. Looking at the code, this is designed this way:
static int
dpif_netlink_port_add__(struct dpif_netlink *dpif, struct netdev *netdev,
odp_port_t *port_nop)
{
...
if (dpif->handlers) {
socksp = vport_create_socksp(dpif, &error);
...
}
...
if (socksp) {
error = vport_add_channels(dpif, *port_nop, socksp);
...
}
...
}
static int
vport_add_channels(struct dpif_netlink *dpif, odp_port_t port_no,
struct nl_sock **socksp)
{
...
uint32_t port_idx = odp_to_u32(port_no);
...
... realloc dpif->handler[i]->channels if needed
...
for (i = 0; i < dpif->n_handlers; i++) {
...
dpif->handlers[i].channels[port_idx].sock = socksp[i];
...
}
...
}
I don't see the motivation behind that. Changing it will be a substantial effort, though.
We are currently trying this workaround to increase the number of file descriptors for ovs-vswitchd --- the following is from my lab. I'm getting back as soon as I have the customer's answer.
++++++++++++++++++++++++++++
Can we try to increase the number of file descriptors for ovs-vswitchd significantly:
~~~
mkdir /etc/systemd/system/ovs-vswitchd.service.d/
echo -e "[Service]\nLimitNOFILE=262144" > /etc/systemd/system/ovs-vswitchd.service.d/limit_no_file.conf
systemctl daemon-reload
systemctl restart openvswitch
~~~
After this change, `systemctl status ovs-vswitchd` should show:
~~~
systemctl status ovs-vswitchd
(...)
Drop-In: /etc/systemd/system/ovs-vswitchd.service.d
└─limit_no_file.conf
(...)
~~~
Finally, verify the max open files for the ovs-vswitchd process:
~~~
[root@undercloud-8 ~]# cat /proc/`pidof ovs-vswitchd`/limits | grep 'Max open'
Max open files 262144 262144 files
~~~
I kept thinking about this over the weekend. Regardless of the ovs bug (or misdesign?) I found on Friday, we should clarify one thing: why there are so many internal ports in ovsdb? Surely there could have never been a point in time where all of those ports were operating together (because of, obviously, not enough file descriptors). Is it because in the past, there were 1868 VMs running (or, possibly, 934 or another fraction of that number depending on how many ports are needed for a single VM) and OpenStack tried to add one more VM? And when it failed, it retried, etc. ad infinitum? Or what is the cause? (In reply to Jiri Benc from comment #13) > I kept thinking about this over the weekend. Regardless of the ovs bug (or > misdesign?) I found on Friday, we should clarify one thing: why there are so > many internal ports in ovsdb? Surely there could have never been a point in > time where all of those ports were operating together (because of, > obviously, not enough file descriptors). > > Is it because in the past, there were 1868 VMs running (or, possibly, 934 or > another fraction of that number depending on how many ports are needed for a > single VM) and OpenStack tried to add one more VM? And when it failed, it > retried, etc. ad infinitum? Or what is the cause? That's a great question and we definitely need to find the answer for it. Thanks for your analysis Jiri, I learnt about how file descriptors are consumed by OVS and how to tweak it. Regarding your question and without having looked into the sosreports yet, the other day we hit some bug that whenever a VM was created, for some unknown reason, two neutron ports were created (even though it had only one NIC). Apparently, this bug is rare and was observed in the past but we weren't able to collect logs. I wonder whether this environment could be experiencing the same issue... As I'm a bit late to the party, we could try to sync tomorrow and I'll check neutron/nova logs to see if I can find a lead. I've been trying to categorize the ports in the system: * Total ports are 4965: $ cat ovs-vsctl_-t_5_show | grep Port | wc -l 4965 * qr ports are 260: $ cat ovs-vsctl_-t_5_show | grep Port | grep "qr-" | wc -l 260 * qg ports are 260: $ cat ovs-vsctl_-t_5_show | grep Port | grep "qg-" | wc -l 263 * tap ports (DHCP) are 3312: $ cat ovs-vsctl_-t_5_show | grep Port | grep "tap" | wc -l 3312 * ha ports are 1107: $ cat ovs-vsctl_-t_5_show | grep Port | grep "ha-" | wc -l 1107 * vxlan ports are 15: $ cat ovs-vsctl_-t_5_show | grep Port | grep "vxlan" | wc -l 15 The most ports are tap ports (DHCP) but in the sosreport I can see fewer namespaces as per this command: $ cat ip_netns | grep qdhcp | wc -l 192 $ cat ip_netns | grep qrouter | wc -l 11 In neutron logs I can see lots of issues connecting to rabbit. Also, DHCP agent log is full of errors (ovsdb timeouts, cannot find device tapXXX, etc.). Questions on my side: - How many networks, tenants and ha routers do they have? - How's dhcp configured? Is it active/active? Thanks, Daniel Sharing privately customer reply Hi,
Let me please clarify the instructions I'd suggest to fix the current issue is:
1. Run ovs-cleanup the script this way:
1.1 Create cleanup.conf file which looks like this inside /etc/neutron/conf.d/neutron-ovs-cleanup:
[DEFAULT]
ovs_vsctl_timeout=1800
1.2 Execute the ovs cleanup script with debug enabled:
/usr/bin/neutron-ovs-cleanup --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-ovs-cleanup --log-file /var/log/neutron/ovs-cleanup.log --debug
At this point we expect that unused ovs devices are gone and unused namespaces are empty.
2. Run netns-cleanup with debug and *without* --force:
/usr/bin/neutron-netns-cleanup --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-netns-cleanup --log-file /var/log/neutron/netns-cleanup.log --debug
At this point we expect all the empty namespaces to be deleted.
3. Enable debug in /etc/neutron/neutron.conf and restart neutron services.
Also fix the logrotate issue because with debug enabled, log files will be way heavier so it's good that we rotate them.
Please, we note that this is providing a sort of workaround because of something that failed before and made those so many ports to be left behind. Also, the ovs-cleanup script didn't work properly which may have helped as in logs we can see that it was run but failed. For us to be able to identify the root cause if it happens again, we suggest to leave debug enabled and monitor the number of ports for a certain amount of time.
Please, let us know your thoughts and/or questions.
The native ovsdb implementation is sometimes slower when deleting lots of ports. Another thing to try adding to Daniel's comment #19 /etc/neutron/conf.d/neutron-ovs-cleanup file is [OVS] ovsdb_interface=vsctl and see if it runs considerably faster. I'm currently investigating the performance differences between the two. It would be good to see if the vsctl interface is faster in a real-world scenario. If so, I need to invest time in optimizing the the native interface in this case. Sent an RFC to the ovs-dev mailing list: https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343508.html Created attachment 1452875 [details]
patch for the kernel data path
Created attachment 1454715 [details]
beta patch for the kernel data path
Maybe this can be approached from the opposite end? How is the fairness ensured with the current solution? There's a finite number of threads reading from a much higher number of sockets. How is the socket to be processed next selected in the user space? Is the fairness really guaranteed? How do the ovs user space locks affect the processing? If there's a code to ensure fairness, it can probably be replicated on the kernel side. After all, we do have a serializing point currently, as we go from a high number of sockets to a low number of threads. We're just moving this point a little bit down to the kernel. (In reply to Jiri Benc from comment #70) > If there's a code to ensure fairness, it can probably be replicated on the > kernel side. My understanding is that ovs-switchd opens a huge number of sockets and polls them, and this results in a (rather random, but somewhat fair) round-robin fashion. The proposals outlined above are already an attempt at replicating this. RFC upstream patch at:
https://patchwork.ozlabs.org/patch/939390/
Created attachment 1477839 [details]
tool to change the datapath round robin algorithm
Created attachment 1477840 [details]
beta patch for the kernel data path
Created attachment 1483331 [details] patch for userspace vswitchd I wrote a tool[1] which calculates the time that a packet sent to a veth outside the bridge needs to do all the travel through the enslaved peer, and then to the bridge passing by the slow path. Before the patch: root@wsfd-netdev87:~# nproc 56 root@wsfd-netdev87:~# ovs-vsctl show |grep -c Port 2223 root@wsfd-netdev87:~# lsof -p $(pidof ovs-vswitchd) |grep -c GENERIC 91187 Idle system root@wsfd-netdev87:~# for i in {1..22}; do echo -n "Port$i: "; weed veth-ext$i ovs_upc_br -i10u -c 10k |awk -F/ '/^eed/{print$4}'; done Port1: 1020 Port2: 1011 Port3: 1010 Port4: 1035 Port5: 1190 Port6: 1019 Port7: 1027 Port8: 967 Port9: 996 Port10: 986 Port11: 1090 Port12: 1004 Port13: 1070 Port14: 978 Port15: 971 Port16: 1023 Port17: 1003 Port18: 1007 Port19: 996 Port20: 1023 Port21: 961 Port22: 1018 System under heavy load root@wsfd-netdev87:~# upc_test.sh start root@wsfd-netdev87:~# weed veth-ext21 ovs_upc_br -c100 -v eed: 7938 ns eed: 8183 ns eed: 6660 ns eed: 7585 ns eed: 7327 ns 100 packets transmitted, 5 received, 95% packet loss eed min/avg/max = 6660/7538/8183 ns With the patch: root@wsfd-netdev87:~# nproc 56 root@wsfd-netdev87:~# ovs-vsctl show |grep -c Port 2223 root@wsfd-netdev87:~# lsof -p $(pidof ovs-vswitchd) |grep -c GENERIC 2227 Idle system root@wsfd-netdev87:~# for i in {1..22}; do echo -n "Port$i: "; weed veth-ext$i ovs_upc_br -i10u -c 10k |awk -F/ '/^eed/{print$4}'; done Port1: 917 Port2: 923 Port3: 844 Port4: 908 Port5: 908 Port6: 859 Port7: 884 Port8: 935 Port9: 856 Port10: 848 Port11: 850 Port12: 874 Port13: 937 Port14: 877 Port15: 907 Port16: 910 Port17: 910 Port18: 887 Port19: 847 Port20: 918 Port21: 809 Port22: 823 System under heavy load root@wsfd-netdev87:~# upc_test.sh start root@wsfd-netdev87:~# weed veth-ext21 ovs_upc_br -c100 -v eed: 4213 ns eed: 3851 ns eed: 3585 ns eed: 3155 ns eed: 4310 ns eed: 4043 ns eed: 3662 ns eed: 4088 ns eed: 3227 ns eed: 4697 ns eed: 4077 ns eed: 3844 ns eed: 4349 ns eed: 3460 ns eed: 4313 ns eed: 3505 ns eed: 4216 ns eed: 4461 ns eed: 3212 ns eed: 3037 ns eed: 4249 ns eed: 3336 ns eed: 2923 ns eed: 4559 ns eed: 4665 ns eed: 2920 ns eed: 3744 ns eed: 3390 ns eed: 4635 ns eed: 3965 ns eed: 3568 ns eed: 3708 ns eed: 3525 ns eed: 4154 ns eed: 3090 ns eed: 3696 ns eed: 3329 ns eed: 3844 ns 100 packets transmitted, 38 received, 62% packet loss eed min/avg/max = 2920/3805/4697 ns Regards, Matteo [1] https://github.com/teknoraver/network-tools (In reply to Matteo Croce from comment #79) > Created attachment 1483331 [details] > patch for userspace vswitchd > > I wrote a tool[1] which calculates the time that a packet sent to a veth > outside the bridge needs to do all the travel through the enslaved peer, and > then to the bridge passing by the slow path. How does this measure per-vport fairness, which was the actual concern raised in [1]? [1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343575.html Hi Stefano, I made a better test to mesure fairness. I setup a bridge with 22 ports, and inject traffic in all but one port. Then I start the latency measure on the idle port, to see if the packets are lost or not. This is the test with stock OVS: # weed veth-ext1 ovs_upc_br -i1m -c100k 100000 packets transmitted, 5172 received, 94% packet loss eed min/avg/max/mdev = 1.1/6.1/18.3/2.2 us and this with the userspace patch: # weed veth-ext1 ovs_upc_br -i1m -c100k 100000 packets transmitted, 95032 received, 4% packet loss eed min/avg/max/mdev = 1.0/2.1/344.5/1.2 us Patch submitted upstream https://mail.openvswitch.org/pipermail/ovs-dev/2018-September/352251.html (In reply to Matteo Croce from comment #81) > Hi Stefano, > > I made a better test to mesure fairness. I setup a bridge with 22 ports, and > inject traffic in all but one port. > > Then I start the latency measure on the idle port, to see if the packets are > lost or not. > > This is the test with stock OVS: > > # weed veth-ext1 ovs_upc_br -i1m -c100k > 100000 packets transmitted, 5172 received, 94% packet loss > eed min/avg/max/mdev = 1.1/6.1/18.3/2.2 us > > and this with the userspace patch: > > # weed veth-ext1 ovs_upc_br -i1m -c100k > 100000 packets transmitted, 95032 received, 4% packet loss > eed min/avg/max/mdev = 1.0/2.1/344.5/1.2 us As I mentioned on IRC: you are keeping 21 ovs-vswitchd threads busy out of 56 CPU threads (hence, 56 possible ovs-vswitchd threads) of your test host. I think you should rather saturate ovs-vswitchd threads to have valid results on this test. Otherwise, ovs-vswitchd can just handle the "control" traffic on a thread which isn't loaded at all. You could e.g. limit the number of CPU threads ovs-vswitchd can run on with taskset(1). But let's see, maybe it's good enough for upstream. (In reply to Matteo Croce from comment #82) > Patch submitted upstream > > https://mail.openvswitch.org/pipermail/ovs-dev/2018-September/352251.html I'm not sure where you #define EPOLLEXCLUSIVE, but it looks like you simply don't use that flag if it's not available (say, Linux kernel versions < 4.5, RHEL versions < 7.3) without keeping per-thread sockets, which, according to commit message, would lead to duplicate events. But perhaps this is also acceptable for upstream. Hi Stefano, I don't #define EPOLLEXCLUSIVE myself it's defined in /usr/include/sys/epoll.h. Right, with older kernels than 4.5 or RHEL 7.3, I just don't use the flag. This of course leads to duplicate events, but even without the patch, many events are duplicated, probably for the way the hashing of the port id in the kernel works. This can be observed by adding a printf in dpif_netlink_recv__(). This seems not to be an issue, because even if multiple threads are waken, only one can read the packet, so there are no duplicate packets, this causes just unnecessary wake up of the handler threads. With the patch and EPOLLEXCLUSIVE instead, exactly only one thread is waken up, leading to lower latency. Cheers, Patch merged upstream: https://mail.openvswitch.org/pipermail/ovs-dev/2018-September/352402.html Reproduce with openvswitch-2.9.0-56.el7fdp.x86_64 [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 1 10 11 12 13 14 15 16 17 2 3 4 5 6 7 8 9 [root@dell-per730-04 ~]# ovs-vsctl add-br ovsbr0 [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 101 105 109 112 116 12 123 127 130 134 138 141 145 149 152 156 16 2 23 27 30 34 38 41 45 49 52 56 6 63 67 70 74 78 81 85 89 92 96 1 102 106 11 113 117 120 124 128 131 135 139 142 146 15 153 157 17 20 24 28 31 35 39 42 46 5 53 57 60 64 68 71 75 79 82 86 9 93 97 10 103 107 110 114 118 121 125 129 132 136 14 143 147 150 154 158 18 21 25 29 32 36 4 43 47 50 54 58 61 65 69 72 76 8 83 87 90 94 98 100 104 108 111 115 119 122 126 13 133 137 140 144 148 151 155 159 19 22 26 3 33 37 40 44 48 51 55 59 62 66 7 73 77 80 84 88 91 95 99 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p0 -- set interface p0 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 102 107 111 116 120 125 13 134 139 143 148 152 157 161 166 170 175 18 184 189 193 22 27 31 36 40 45 5 54 59 63 68 72 77 81 86 90 95 1 103 108 112 117 121 126 130 135 14 144 149 153 158 162 167 171 176 180 185 19 194 23 28 32 37 41 46 50 55 6 64 69 73 78 82 87 91 96 10 104 109 113 118 122 127 131 136 140 145 15 154 159 163 168 172 177 181 186 190 2 24 29 33 38 42 47 51 56 60 65 7 74 79 83 88 92 97 100 105 11 114 119 123 128 132 137 141 146 150 155 16 164 169 173 178 182 187 191 20 25 3 34 39 43 48 52 57 61 66 70 75 8 84 89 93 98 101 106 110 115 12 124 129 133 138 142 147 151 156 160 165 17 174 179 183 188 192 21 26 30 35 4 44 49 53 58 62 67 71 76 80 85 9 94 99 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p1 -- set interface p1 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 103 109 114 12 125 130 136 141 147 152 158 163 169 174 18 185 190 196 200 206 211 217 222 228 27 32 38 43 49 54 6 65 70 76 81 87 92 98 1 104 11 115 120 126 131 137 142 148 153 159 164 17 175 180 186 191 197 201 207 212 218 223 229 28 33 39 44 5 55 60 66 71 77 82 88 93 99 10 105 110 116 121 127 132 138 143 149 154 16 165 170 176 181 187 192 198 202 208 213 219 224 23 29 34 4 45 50 56 61 67 72 78 83 89 94 100 106 111 117 122 128 133 139 144 15 155 160 166 171 177 182 188 193 199 203 209 214 22 225 24 3 35 40 46 51 57 62 68 73 79 84 9 95 101 107 112 118 123 129 134 14 145 150 156 161 167 172 178 183 189 194 2 204 21 215 220 226 25 30 36 41 47 52 58 63 69 74 8 85 90 96 102 108 113 119 124 13 135 140 146 151 157 162 168 173 179 184 19 195 20 205 210 216 221 227 26 31 37 42 48 53 59 64 7 75 80 86 91 97 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p2 -- set interface p2 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 103 109 114 12 125 130 136 141 147 152 158 163 169 174 18 185 190 196 200 206 211 217 222 228 233 239 244 25 255 260 28 33 39 44 5 55 60 66 71 77 82 88 93 99 1 104 11 115 120 126 131 137 142 148 153 159 164 17 175 180 186 191 197 201 207 212 218 223 229 234 24 245 250 256 261 29 34 4 45 50 56 61 67 72 78 83 89 94 10 105 110 116 121 127 132 138 143 149 154 16 165 170 176 181 187 192 198 202 208 213 219 224 23 235 240 246 251 257 262 3 35 40 46 51 57 62 68 73 79 84 9 95 100 106 111 117 122 128 133 139 144 15 155 160 166 171 177 182 188 193 199 203 209 214 22 225 230 236 241 247 252 258 263 30 36 41 47 52 58 63 69 74 8 85 90 96 101 107 112 118 123 129 134 14 145 150 156 161 167 172 178 183 189 194 2 204 21 215 220 226 231 237 242 248 253 259 264 31 37 42 48 53 59 64 7 75 80 86 91 97 102 108 113 119 124 13 135 140 146 151 157 162 168 173 179 184 19 195 20 205 210 216 221 227 232 238 243 249 254 26 27 32 38 43 49 54 6 65 70 76 81 87 92 98 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p3 -- set interface p3 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 104 110 117 123 13 136 142 149 155 161 168 174 180 187 193 2 205 211 218 224 230 237 243 25 256 262 269 275 281 288 294 30 37 43 5 56 62 69 75 81 88 94 1 105 111 118 124 130 137 143 15 156 162 169 175 181 188 194 20 206 212 219 225 231 238 244 250 257 263 27 276 282 289 295 31 38 44 50 57 63 7 76 82 89 95 10 106 112 119 125 131 138 144 150 157 163 17 176 182 189 195 200 207 213 22 226 232 239 245 251 258 264 270 277 283 29 296 32 39 45 51 58 64 70 77 83 9 96 100 107 113 12 126 132 139 145 151 158 164 170 177 183 19 196 201 208 214 220 227 233 24 246 252 259 265 271 278 284 290 297 33 4 46 52 59 65 71 78 84 90 97 101 108 114 120 127 133 14 146 152 159 165 171 178 184 190 197 202 209 215 221 228 234 240 247 253 26 266 272 279 285 291 298 34 40 47 53 6 66 72 79 85 91 98 102 109 115 121 128 134 140 147 153 16 166 172 179 185 191 198 203 21 216 222 229 235 241 248 254 260 267 273 28 286 292 299 35 41 48 54 60 67 73 8 86 92 99 103 11 116 122 129 135 141 148 154 160 167 173 18 186 192 199 204 210 217 223 23 236 242 249 255 261 268 274 280 287 293 3 36 42 49 55 61 68 74 80 87 93 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p4 -- set interface p4 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 105 112 12 127 134 141 149 156 163 170 178 185 192 2 206 213 220 228 235 242 25 257 264 271 279 286 293 30 307 314 321 329 35 42 5 57 64 71 79 86 93 1 106 113 120 128 135 142 15 157 164 171 179 186 193 20 207 214 221 229 236 243 250 258 265 272 28 287 294 300 308 315 322 33 36 43 50 58 65 72 8 87 94 10 107 114 121 129 136 143 150 158 165 172 18 187 194 200 208 215 222 23 237 244 251 259 266 273 280 288 295 301 309 316 323 330 37 44 51 59 66 73 80 88 95 100 108 115 122 13 137 144 151 159 166 173 180 188 195 201 209 216 223 230 238 245 252 26 267 274 281 289 296 302 31 317 324 331 38 45 52 6 67 74 81 89 96 101 109 116 123 130 138 145 152 16 167 174 181 189 196 202 21 217 224 231 239 246 253 260 268 275 282 29 297 303 310 318 325 332 39 46 53 60 68 75 82 9 97 102 11 117 124 131 139 146 153 160 168 175 182 19 197 203 210 218 225 232 24 247 254 261 269 276 283 290 298 304 311 319 326 333 4 47 54 61 69 76 83 90 98 103 110 118 125 132 14 147 154 161 169 176 183 190 198 204 211 219 226 233 240 248 255 262 27 277 284 291 299 305 312 32 327 334 40 48 55 62 7 77 84 91 99 104 111 119 126 133 140 148 155 162 17 177 184 191 199 205 212 22 227 234 241 249 256 263 270 278 285 292 3 306 313 320 328 34 41 49 56 63 70 78 85 92 [root@dell-per730-04 ~]# Verify with openvswitch-2.9.0-70.el7fdp.x86_64 [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 1 10 11 12 13 14 15 16 17 2 3 4 5 6 7 8 9 [root@dell-per730-04 ~]# ovs-vsctl add-br ovsbr0 [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 47 49 50 52 54 56 58 6 61 63 65 67 69 70 72 74 76 78 8 81 83 85 87 89 90 1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 48 5 51 53 55 57 59 60 62 64 66 68 7 71 73 75 77 79 80 82 84 86 88 9 91 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p0 -- set interface p0 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 47 49 50 52 54 56 58 6 61 63 65 67 69 70 72 74 76 78 8 81 83 85 87 89 90 92 1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 48 5 51 53 55 57 59 60 62 64 66 68 7 71 73 75 77 79 80 82 84 86 88 9 91 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p1 -- set interface p1 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 47 49 50 52 54 56 58 6 61 63 65 67 69 70 72 74 76 78 8 81 83 85 87 89 90 92 1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 48 5 51 53 55 57 59 60 62 64 66 68 7 71 73 75 77 79 80 82 84 86 88 9 91 93 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p2 -- set interface p2 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 47 49 50 52 54 56 58 6 61 63 65 67 69 70 72 74 76 78 8 81 83 85 87 89 90 92 94 1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 48 5 51 53 55 57 59 60 62 64 66 68 7 71 73 75 77 79 80 82 84 86 88 9 91 93 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p3 -- set interface p3 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 47 49 50 52 54 56 58 6 61 63 65 67 69 70 72 74 76 78 8 81 83 85 87 89 90 92 94 1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 48 5 51 53 55 57 59 60 62 64 66 68 7 71 73 75 77 79 80 82 84 86 88 9 91 93 95 [root@dell-per730-04 ~]# ovs-vsctl add-port ovsbr0 p4 -- set interface p4 type=internal [root@dell-per730-04 ~]# ls /proc/`pidof ovs-vswitchd`/fd 0 10 12 14 16 18 2 21 23 25 27 29 30 32 34 36 38 4 41 43 45 47 49 50 52 54 56 58 6 61 63 65 67 69 70 72 74 76 78 8 81 83 85 87 89 90 92 94 96 1 11 13 15 17 19 20 22 24 26 28 3 31 33 35 37 39 40 42 44 46 48 5 51 53 55 57 59 60 62 64 66 68 7 71 73 75 77 79 80 82 84 86 88 9 91 93 95 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3500 |