Bug 2134880
| Summary: | ovn-controller forgets to install flows upon OFPBFC_TIMEOUT | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | François Rigault <frigo> |
| Component: | ovn-2021 | Assignee: | Dumitru Ceara <dceara> |
| Status: | CLOSED NEXTRELEASE | QA Contact: | Jianlin Shi <jishi> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | RHEL 8.0 | CC: | ctrautma, dcbw, dceara, fgrosjea, i.maximets, jiji, milti, mleonard, mmichels, sdodson |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-06-05 08:06:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
also reproducing on main branch Thanks for the bug report and thanks for the detailed reproducer. I've marked triage completed since we have the information we need to pursue this. for completeness
> installing over 50k flows
table 44 contain flows with more than 1200 conjunctions, totaling over 600k conjunction actions. During processing ovn-controller gets stuck.
(In reply to François Rigault from comment #3) > for completeness > > installing over 50k flows > table 44 contain flows with more than 1200 conjunctions, totaling over 600k > conjunction actions. During processing ovn-controller gets stuck. Would you be able to attach the OVN SB and OVS flow dumps as private attachments? If not that's OK, but it may help. I think the workaround (to run 'ovs-vsctl set o .
other_config:bundle-idle-timeout=100') is the best option for now.
The problem is that (in valid scenarios) ovn-controller might take long
to process SB DB changes. E.g., if there are lots of ACLs that generate
lots of conjunctive openflows:
ACL: "ip4.src == {$AS1} && tcp && X<=tcp.dst<=Y && outport == @PG2" allow-related
In the case above, depending on how many such ACLs refer to @PG2,
ovn-controller might have to update a reasonably large number of
openflows.
The rconn connection (ovn-controller to ovs-vswitchd) is basically a
unix pipe but may get backlogged. ovn-controller sends all updates
generated by an event loop iteration as a single bundle, that is:
1. bundle-open msg
2. update messages for the flows that need to change
3. bundle-commit msg
Overly simplified, the way ovn-controller currently works is:
a. get updates from SB
b. process these updates and generate "desired" openflows
c. if rconn is not backlogged send openflows to ovs-vswitchd,
as a bundle, synchronizing "desired" state to "installed" state.
Focusing on two iterations, the sequence of events may be:
-----------
Iteration1:
-----------
a. get updates from SB
b. process these updates and generate "desired" openflows (lets assume
this generates quite a lot of desired openflow modifications)
c.1. add bundle-open msg to rconn
c.2. add openflow mod msgs to rconn (only some of these make it through,
the rest gets queued, the rconn is backlogged at this point).
c.3. add bundle-commit msg to rconn (this gets queued)
----------
Iteration2:
-----------
a. get updates from SB (rconn is still backlogged)
b. process the updates and generate "desired" openflows (lets assume
this takes 10+ seconds for the specific SB updates) << at some point while this step was running OVS declared the bundle operation timeout!
As reported, ovn-controller ignores the timeout reply. But
unfortunately we can't just automatically act on that and immediately
trigger a recompute (fixed so that it actually clears and installs
all flows). That's because it's possible that this will generate even
more backlog on the rconn connection.
Another option (suggested by Ilya) could be to skip the incremental
processing engine run as long as there are messages queued on the
rconn (rconn is backlogged). This would ensure that the bundle-commit
is sent as soon as reasonably possible.
Thanks,
Dumitru
(In reply to Dumitru Ceara from comment #5) > As reported, ovn-controller ignores the timeout reply. But > unfortunately we can't just automatically act on that and immediately > trigger a recompute (fixed so that it actually clears and installs > all flows). That's because it's possible that this will generate even > more backlog on the rconn connection. > It turns out this is not completely true. If we trigger an OF reconnect that will flush the backlog. Patch posted upstream for review: https://patchwork.ozlabs.org/project/ovn/list/?series=356592&state=* ovn23.06 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211291 ovn23.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211292 ovn23.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211293 ovn23.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211294 ovn22.12 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211295 ovn22.12 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211296 ovn22.03 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211298 ovn22.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2211299 Will be fixed starting with ovn22.03. |
Description of problem: on some of our deployments with ovn running on VM and installing over 50k flows, we find ovn-controller not installing some flows. ovn-controller is aware of the flow but not installing them. Version-Release number of selected component (if applicable): ovn branch-21.12 from sources ovn-2021-21.12.0-58.el8fdp How reproducible: in sandbox, using gdb to pause ovn the cause seems to be that ovn-controller consider flows "installed" even though openvswitch replied with a timeout. I don't know the exact pattern, so it takes some retries before reproducing it. Goal is to get messages like this sending OFPBFC_TIMEOUT error reply to OFPT_BUNDLE_ADD_MESSAGE message Steps to Reproduce: 1. we prepare some minimal set-up ovs-vsctl set o . other_config:bundle-idle-timeout=1 ovn-nbctl ls-add fnet ovn-nbctl lsp-add fnet fr1p ovn-nbctl lsp-set-addresses fr1p "fa:16:00:00:00:11 192.168.0.11" ovn-nbctl lsp-set-enabled fr1p enabled ovn-nbctl lsp-set-port-security fr1p "fa:16:00:00:00:11 192.168.0.11 192.168.0.5" ovs-vsctl add-port br-int p1 -- \ set Interface p1 external_ids:iface-id=fr1p -- \ set Interface p1 type=internal ip netns add n1 ip link set p1 netns n1 ip netns exec n1 ip link set dev p1 address fa:16:00:00:00:11 ip netns exec n1 ip addr add 192.168.0.11/28 dev p1 ip netns exec n1 ip link set dev p1 up 2. we count the number of flows when this runs properly: ovn-nbctl acl-del fnet ovn-nbctl acl-add fnet to-lport 1002 '(outport == "fr1p" && ip && icmp)' allow-related ovs-ofctl dump-flows br-int | wc -l # I get 196 3. we attach gdb on a different terminal gdb -p `pgrep ovn-controller` br rconn_send cont # when ovn-nbctl acl-add is running, we run a couple of times fin p sleep(1) cont in order to trigger the timeout above. Actual results: ovs-ofctl dump-flows br-int | wc -l output varies: sometimes we get more, sometimes we get less ovn-appctl inc-engine/recompute does not repair the flows, they stay like that until ovn-controller is restarted. Expected results: we should have the same number of flows even when ovn-controller froze inc-engine/recompute should take care of recomputing flows Additional info: we can workaround the issue by setting ovs-vsctl set o . other_config:bundle-idle-timeout=100 or restarting ovn-controller. Initially discovered because flows handling arp were missing, causing small network outages.