Description of problem: Consider the following topology: 100.64.0.1 10.244.0.6 ext--GR1 ------------OVN cluster router (DR)-----------pod1 | ext--GR2 ------------- 100.64.0.2 pod1 and GR 1 are on the same node. GR1 and GR2 both SNAT ingress traffic to them. OVN DR has static routes for: 10.244.0.0/24 via 100.64.0.1 src-ip Now if a packet comes from GR1 from external network (ext) it will be SNAT to 100.64.0.1 and send to pod1. pod 1 sends a reply packet and it goes back to 100.64.0.1 correctly. However, if a packet ingresses at GR2, it gets SNAT to 100.64.0.2 and the packet is sent to pod 1. pod1 then replies and the packet goes to the DR. At this point OVN uses the static route and incorrectly sends the packet to GR1 (asymmetrical routing). The correct behavior should have been to send the packet to GR2. This is because the destination is 100.64.0.2, and a router should always have a lower metric for directly connected routes. A workaround to this problem is to add a dummy static route with higher prefix match like: 100.64.0.1/32 via 100.64.0.1 dst-ip 100.64.0.2/32 via 100.64.0.2 dst-ip ^This is kind of silly, but it works.
Here are the relevant lflows: table=10(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 100.64.0.2/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 100.64.0.3/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.3; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 100.64.0.4/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.4; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 10.244.0.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 10.244.0.1; eth.src = 0a:58:0a:f4:00:01; outport = "rtos-ovn-worker"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 10.244.1.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 10.244.1.1; eth.src = 0a:58:0a:f4:01:01; outport = "rtos-ovn-control-plane"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 10.244.2.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 10.244.2.1; eth.src = 0a:58:0a:f4:02:01; outport = "rtos-ovn-worker2"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=48 , match=(ip4.src == 10.244.0.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.4; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=48 , match=(ip4.src == 10.244.1.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=48 , match=(ip4.src == 10.244.2.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.3; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=41 , match=(ip4.dst == 169.254.0.0/20), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 169.254.0.2; eth.src = 0a:58:a9:fe:00:02; outport = "rtos-node_local_switch"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=33 , match=(ip4.dst == 100.64.0.0/16), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) The priority 65 flows here are the outcome of our workaround routes: table=10(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 100.64.0.2/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 100.64.0.3/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.3; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) table=10(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 100.64.0.4/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.4; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) 100.64.0.1/32 via 100.64.0.1 dst-ip 100.64.0.2/32 via 100.64.0.2 dst-ip The directly connected route has a priority of 33 here: table=10(lr_in_ip_routing ), priority=33 , match=(ip4.dst == 100.64.0.0/16), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 100.64.0.1; eth.src = 0a:58:64:40:00:01; outport = "rtoj-ovn_cluster_router"; flags.loopback = 1; next;) As Lorenzo explained, this is calculated by prefix * 2 + 1. However, this route should have the highest possible priority (67 should be safe). This is because this is a directly connected route which has a lower metric in routing than any other static route. On most real routing appliances a directly connected route has a metric of 0, and a user is not able to add a static route with a metric any lower than 1. In other words, directly connected routes *always* have higher priority and cannot be overidden with static routes. If a user wants to do something different than traditional routing, they may use policy routing (available in OVN) to override the route lookup determined.
(In reply to Tim Rozet from comment #0) > Description of problem: > Consider the following topology: > > 100.64.0.1 10.244.0.6 > ext--GR1 ------------OVN cluster router (DR)-----------pod1 > | > ext--GR2 ------------- > 100.64.0.2 > > pod1 and GR 1 are on the same node. GR1 and GR2 both SNAT ingress traffic to > them. > > OVN DR has static routes for: > > 10.244.0.0/24 via 100.64.0.1 src-ip > > Now if a packet comes from GR1 from external network (ext) it will be SNAT > to 100.64.0.1 and send to pod1. pod 1 sends a reply packet and it goes back > to 100.64.0.1 correctly. However, if a packet ingresses at GR2, it gets > SNAT to 100.64.0.2 and the packet is sent to pod 1. pod1 then replies and > the packet goes to the DR. At this point OVN uses the static route and > incorrectly sends the packet to GR1 (asymmetrical routing). > > The correct behavior should have been to send the packet to GR2. This is > because the destination is 100.64.0.2, and a router should always have a > lower metric for directly connected routes. A workaround to this problem is > to add a dummy static route with higher prefix match like: > > 100.64.0.1/32 via 100.64.0.1 dst-ip > 100.64.0.2/32 via 100.64.0.2 dst-ip > > ^This is kind of silly, but it works. Based on the upstream discussion [0], the issue can be fixed using the policy routing table implemented in OVN without removing the capability to overwrite automatic routes with static ones according to the "longest prefix match". @Tim: do you agree to close the bz as "not a bug"? [0] https://mail.openvswitch.org/pipermail/ovs-dev/2021-February/380295.html
Yeah, we can just keep the workaround for now. It doesn't hurt us to have it.