Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2224199

Summary: ovn-controller replace CT zone UUID names with LR/LS names
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Surya Seetharaman <surya>
Component: ovn23.09Assignee: Ales Musil <amusil>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: FDP 23.ACC: amusil, ctrautma, dcbw, dceara, jiji, mmichels, ovn-bot
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn23.09-23.09.0-alpha.102.el9fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-01-24 11:17:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Surya Seetharaman 2023-07-20 07:35:00 UTC
Description of problem:

OVNK ran into an issue while upgrading from non interconnect to interconnect model. During the upgrade as we move to the new zone, new NB and SB dbs are created which will have new UUIDs compared to the older state but the elements are the same (name is the same). Since the UUIDs are different, controller on start up flushes zone 0 which is used by GR's on all nodes in OVNK for doing SNAT. Thus ovn-controller thinks it also owns zone0. On the host, we use zone 0 by default for conntrack and flushing all entries causes outage for existing traffic. One such scenario is:

client (10.0.128.4) is talking to server (10.0.0.5) via GCP load balancer (10.0.0.2 is the VIP that is dnated to 10.0.0.5 - this has NO ovn lb involvement); the server now moves to remote zone, controller on server flushes zone0 which causes entries to be flushed, now if the server is reinitiating the retransmit first it has no idea of history and now creates an entry with srcIP that of 10.0.0.5 and it doesn't know about the older entry 10.0.0.2 which is supposed to be used as the srcIP and this breaks traffic resulting in drop at the server node's interface and after 20-30 seconds of timeout; the client sends the reset which causes the srcPort to change and new traffic to start flowing and only then things settle down. TL;DR flushing zone0 causes traffic outage for 40-50 seconds which is bad.

correct entry before flush: 
tcp 6 431999 ESTABLISHED src=10.0.128.4 dst=10.0.0.2 sport=46122 dport=6443 src=10.0.0.5 dst=10.0.128.4 sport=6443 dport=36363 [ASSURED] mark=0 secctx=system_u:object_r
:unlabeled_t:s0 use=1

bad entry after flush because 10.0.0.5 initiates the connection first: so it thinks its just directly talking to the other node

tcp 6 431999 ESTABLISHED src=10.0.128.4 dst=10.0.0.5 sport=46122 dport=6443 src=10.0.0.5 dst=10.0.128.4 sport=6443 dport=36363 [ASSURED] mark=0 secctx=system_u:object_r
:unlabeled_t:s0 use=1


See previous bug related bug: https://bugzilla.redhat.com/show_bug.cgi?id=2087194

We need to revisit OVN owning the zone0.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Dumitru Ceara 2023-07-20 08:51:40 UTC
FWIW, I think the title is misleading.  There's nothing blocking the host from using *any* CT zone it wants.

IMO ovn-controller should flush all zones it uses.  There is one special case, when ovn-controller shares a zone with the host.  That's normally when LR.options:snat-ct-zone=<ZONE> is set in the NB.  Only in that case it *might* be acceptable to not flush the zone, if the CMS explicitly requests that.

Comment 4 Dumitru Ceara 2023-07-21 10:52:50 UTC
Summarizing what we discussed offline:

The real underlying problem is that ovn-controller was flushing CT zone 0 during an ovn-kubernetes migration from non-IC to IC deployments.  That's because the SB database was being reconstructed and the datapath associated to the gateway router that had snat-ct-zone=0 set was changing UUID; the ovn-controller mechanism that avoids flushing zones that were already in use by OVN matches on datapath UUIDs and not names so ovn-controller was incorrectly assuming that the logical datapath had changed.

It's not desirable to add more configuration knobs for avoiding CT zone flush (those would have to be per switch/router and would over-complicate the code); instead we can change ovn-controller to try matching both on UUID and on switch/router name when mapping required CT zones to already existing ones.  Once ovn-kubernetes upgrades to a version of OVN that supports both mappings no flush will happen anymore when the SB UUID changes unless the SB datapath name changes too.  That should properly fix any traffic disruption issues caused by conntrack flush when upgrading from non-IC to IC deployments.

Comment 5 OVN Bot 2023-07-28 04:05:22 UTC
ovn23.09 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2227121
ovn23.06 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2227122
ovn23.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2227123

Comment 6 Dan Williams 2023-08-04 13:47:13 UTC
Upstream patch applied: http://patchwork.ozlabs.org/project/ovn/patch/20230726124239.66275-1-amusil@redhat.com/

Comment 7 Mark Michelson 2023-08-04 14:37:05 UTC
*** Bug 2227121 has been marked as a duplicate of this bug. ***

Comment 10 Jianlin Shi 2023-12-05 02:21:04 UTC
with reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2227123#c3

Verified on ovn23.09-23.09.0-73.el9:

[root@kvm-01-guest22 bz2224199]# rpm -qa | grep -E "openvswitch3.1|ovn23.09"
openvswitch3.1-3.1.0-70.el9fdp.x86_64
ovn23.09-23.09.0-73.el9fdp.x86_64
ovn23.09-central-23.09.0-73.el9fdp.x86_64
ovn23.09-host-23.09.0-73.el9fdp.x86_64

+ ovn-nbctl set logical_router lr1 options:chassis=hv1
+ ovn-nbctl lr-nat-add lr1 snat 172.17.1.1 192.168.1.1
+ ovn-nbctl --wait=hv sync
+ ovs-vsctl list bridge
_uuid               : 5df93910-da9a-4805-9875-a616b9f9e7e9
auto_attach         : []
controller          : []
datapath_id         : "00000aa33b1ec712"
datapath_type       : system
datapath_version    : "<unknown>"
external_ids        : {ct-zone-lr1-ls1="10", ct-zone-lr1-ls2="9", ct-zone-lr1_dnat="1", ct-zone-lr1_snat="11", ct-zone-ls1-lr1="8", ct-zone-ls1_dnat="12", ct-zone-ls1_snat="5", ct-zone-ls1p1="3", ct-zone-ls1p2="7", ct-zone-ls2-lr1="6", ct-zone-ls2_dnat="4", ct-zone-ls2_snat="2", ct-zone-ls2p1="13", ovn-nb-cfg="1", ovn-nb-cfg-ts="1701742776848", ovn-startup-ts="1701742766697"}

<=== ct-zone name is related to ls/lr name


fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : br-int
netflow             : []
other_config        : {disable-in-band="true", hwaddr="0a:a3:3b:1e:c7:12"}
ports               : [1c98001b-be02-40c9-960d-a1599d3edc64, 258e41ca-f789-4c4f-862d-1d802075874b, 71938d9c-0784-495b-a803-913768e001a9, fe17825f-f75b-4112-a3f6-cacb412ba3ae]
protocols           : []
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false

Comment 12 errata-xmlrpc 2024-01-24 11:17:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn23.09 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0392