Description of problem: OVNK ran into an issue while upgrading from non interconnect to interconnect model. During the upgrade as we move to the new zone, new NB and SB dbs are created which will have new UUIDs compared to the older state but the elements are the same (name is the same). Since the UUIDs are different, controller on start up flushes zone 0 which is used by GR's on all nodes in OVNK for doing SNAT. Thus ovn-controller thinks it also owns zone0. On the host, we use zone 0 by default for conntrack and flushing all entries causes outage for existing traffic. One such scenario is: client (10.0.128.4) is talking to server (10.0.0.5) via GCP load balancer (10.0.0.2 is the VIP that is dnated to 10.0.0.5 - this has NO ovn lb involvement); the server now moves to remote zone, controller on server flushes zone0 which causes entries to be flushed, now if the server is reinitiating the retransmit first it has no idea of history and now creates an entry with srcIP that of 10.0.0.5 and it doesn't know about the older entry 10.0.0.2 which is supposed to be used as the srcIP and this breaks traffic resulting in drop at the server node's interface and after 20-30 seconds of timeout; the client sends the reset which causes the srcPort to change and new traffic to start flowing and only then things settle down. TL;DR flushing zone0 causes traffic outage for 40-50 seconds which is bad. correct entry before flush: tcp 6 431999 ESTABLISHED src=10.0.128.4 dst=10.0.0.2 sport=46122 dport=6443 src=10.0.0.5 dst=10.0.128.4 sport=6443 dport=36363 [ASSURED] mark=0 secctx=system_u:object_r :unlabeled_t:s0 use=1 bad entry after flush because 10.0.0.5 initiates the connection first: so it thinks its just directly talking to the other node tcp 6 431999 ESTABLISHED src=10.0.128.4 dst=10.0.0.5 sport=46122 dport=6443 src=10.0.0.5 dst=10.0.128.4 sport=6443 dport=36363 [ASSURED] mark=0 secctx=system_u:object_r :unlabeled_t:s0 use=1 See previous bug related bug: https://bugzilla.redhat.com/show_bug.cgi?id=2087194 We need to revisit OVN owning the zone0. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
FWIW, I think the title is misleading. There's nothing blocking the host from using *any* CT zone it wants. IMO ovn-controller should flush all zones it uses. There is one special case, when ovn-controller shares a zone with the host. That's normally when LR.options:snat-ct-zone=<ZONE> is set in the NB. Only in that case it *might* be acceptable to not flush the zone, if the CMS explicitly requests that.
Summarizing what we discussed offline: The real underlying problem is that ovn-controller was flushing CT zone 0 during an ovn-kubernetes migration from non-IC to IC deployments. That's because the SB database was being reconstructed and the datapath associated to the gateway router that had snat-ct-zone=0 set was changing UUID; the ovn-controller mechanism that avoids flushing zones that were already in use by OVN matches on datapath UUIDs and not names so ovn-controller was incorrectly assuming that the logical datapath had changed. It's not desirable to add more configuration knobs for avoiding CT zone flush (those would have to be per switch/router and would over-complicate the code); instead we can change ovn-controller to try matching both on UUID and on switch/router name when mapping required CT zones to already existing ones. Once ovn-kubernetes upgrades to a version of OVN that supports both mappings no flush will happen anymore when the SB UUID changes unless the SB datapath name changes too. That should properly fix any traffic disruption issues caused by conntrack flush when upgrading from non-IC to IC deployments.
ovn23.09 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2227121 ovn23.06 fast-datapath-rhel-8 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2227122 ovn23.06 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2227123
Upstream patch applied: http://patchwork.ozlabs.org/project/ovn/patch/20230726124239.66275-1-amusil@redhat.com/
*** Bug 2227121 has been marked as a duplicate of this bug. ***