Description of problem: Today with an OVN router we have the ability to configure a pool of ecmp routes like: ovn-nbctl --policy=src-ip --ecmp lr-route-add GR_node1 10.0.0.6/32 1.1.1.3 ovn-nbctl --policy=src-ip --ecmp lr-route-add GR_node1 10.0.0.6/32 1.1.1.4 Consider the following external diagram: +------------+ |EX router 1 | External Net |1.1.1.3 | +----------+ +----------+ +----------+ ----| | | 10.0.0.6 | | OVN DR | |OVN GR |1.1.1.1------/ +------------+ | POD |---------------| |-------------| |----/ | | | | | | --------\ +------------+ +----------+ +----------+ +----------+ -------| | | EX router2| | 1.1.1.4 | +------------+ This will allow traffic coming from 10.0.0.6 to be hashed and load-balanced to either 1.1.1.3, or 1.1.1.4. These destinations may be on an external network, or may be ports attached to another OVN node. The only requirement is that these endpoints are in a different subnet, so that traffic *must* flow through an OVN router to be balanced by ecmp. This problem with this setup is that if 1.1.1.3 or 1.1.1.4 have a network outage, there is no way for OVN router to know and remove that endpoint from its ecmp pool. In other routers it is possible to configure bi-directional forwarding detection (BFD) with routing protocols between directly connected neighbors. This is even possible on configured static routes (like shown above). The request is to be able to do this same configuration with an OVN router (GR or DR) to be able to bring up a BFD session with a directly connected neighbor. The configuration would look like this: 1. enable BFD on the L3 interface for the router (typically done on other routers) and specify BFD timers (this is somewhat optional for this use case) 2. enable BFD on the routes programmed into the router: ovn-nbctl --policy=src-ip --ecmp lr-route-add GR_node1 10.0.0.6/32 1.1.1.3 bfd ovn-nbctl --policy=src-ip --ecmp lr-route-add GR_node1 10.0.0.6/32 1.1.1.4 bfd Upon receiving this configuration, OVN would then send BFD messages to 1.1.1.3 and 1.1.1.4. This would require the .3 and .4 endpoints to also have BFD configured and point to the OVN Router IP. Since the OVN "router" is really a logical construct and these are control plane packets, I think ovn-controller will need to handle processing this control traffic. Then somehow when an outage is detected on a peer, openflow needs to no longer use that endpoint in its ecmp group.
ovn bfd support: https://patchwork.ozlabs.org/project/ovn/cover/cover.1606298055.git.lorenzo.bianconi@redhat.com/
verified on version: # rpm -qa|grep ovn ovn2.13-host-20.12.0-15.el8fdp.x86_64 ovn2.13-central-20.12.0-15.el8fdp.x86_64 ovn2.13-20.12.0-15.el8fdp.x86_64 topo: vm1-----network1----router1-------public------external-----pc | vm2 script: ovn-nbctl ls-add network1 ovn-nbctl lsp-add network1 vm1 ovn-nbctl lsp-set-addresses vm1 "40:44:00:00:00:01 192.168.1.11 2000::11" ovn-nbctl lsp-add network1 vm2 ovn-nbctl lsp-set-addresses vm2 "40:44:00:00:00:02 192.168.1.12 2000::12" ovn-nbctl ls-add public ovn-nbctl lsp-add public public-localnet ovn-nbctl lsp-set-type public-localnet localnet ovn-nbctl lsp-set-addresses public-localnet unknown ovn-nbctl lsp-set-options public-localnet network_name=external ovs-vsctl add-br br-labNet ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=external:br-labNet ovs-vsctl add-port br-labNet ha_veth0 ip link set br-labNet up ovn-nbctl lr-add router1 ovn-nbctl lrp-add router1 router1-net1 40:44:00:00:00:04 192.168.1.1/24 2000::1/64 ovn-nbctl lsp-add network1 net1-router1 ovn-nbctl lsp-set-type net1-router1 router ovn-nbctl lsp-set-addresses net1-router1 router ovn-nbctl lsp-set-options net1-router1 router-port=router1-net1 ovn-nbctl lrp-add router1 router1-net2 40:44:00:00:00:05 192.168.2.1/24 2001::1/64 ovn-nbctl lsp-add network2 net2-router1 ovn-nbctl lsp-set-type net2-router1 router ovn-nbctl lsp-set-addresses net2-router1 router ovn-nbctl lsp-set-options net2-router1 router-port=router1-net2 ovn-nbctl lrp-add router1 router1-public 40:44:00:00:00:06 172.16.1.1/24 2002::1/64 ovn-nbctl lsp-add public public-router1 ovn-nbctl lsp-set-type public-router1 router ovn-nbctl lsp-set-addresses public-router1 router ovn-nbctl lsp-set-options public-router1 router-port=router1-public ovn-nbctl --id=@gc0 create Gateway_Chassis name=public-gw1 chassis_name=hv1 priority=20 -- --id=@gc1 create Gateway_Chassis name=public-gw2 chassis_name=hv0 priority=10 -- set Logical_Router_Port router1-public 'gateway_chassis=[@gc0,@gc1]' ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal ip netns add vm1 ip link set vm1 netns vm1 ip netns exec vm1 ip link set lo up ip netns exec vm1 ip link set vm1 up ip netns exec vm1 ip link set vm1 address 40:44:00:00:00:01 ip netns exec vm1 ip addr add 192.168.1.11/24 dev vm1 ip netns exec vm1 ip -6 addr add 2000::11/64 dev vm1 ip netns exec vm1 ip route add default via 192.168.1.1 dev vm1 ip netns exec vm1 ip -6 route add default via 2000:: dev vm1 ovs-vsctl set Interface vm1 external_ids:iface-id=vm1 ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal ip netns add vm2 ip link set vm2 netns vm2 ip netns exec vm2 ip link set lo up ip netns exec vm2 ip link set vm2 up ip netns exec vm2 ip link set vm2 address 40:44:00:00:00:02 ip netns exec vm2 ip addr add 192.168.1.12/24 dev vm2 ip netns exec vm2 ip -6 addr add 192.168.1.12/24 dev vm2 ip netns exec vm2 ip route add default via 192.168.1.1 dev vm2 ip netns exec vm2 ip -6 route add default via 2000:: dev vm2 ovs-vsctl set Interface vm2 external_ids:iface-id=vm2 ip netns add external ip link add ha_veth0 type veth peer name ha_veth0_p netns external ip netns exec external ip link set lo up ip netns exec external ip link set ha_veth0_p up ip link set ha_veth0 up ip netns exec external ip addr add 172.16.1.50/24 dev ha_veth0_p ip netns exec external ip addr add 172.16.1.51/24 dev ha_veth0_p ip netns exec external ip -6 addr add 2002::50/64 dev ha_veth0_p ip netns exec external ip -6 addr add 2002::51/64 dev ha_veth0_p ip link add veth0 type veth peer name veth0_peer ip link set up dev veth0 ip link set veth0_peer netns external ip netns exec external ip link set up dev veth0_peer ip netns exec external ip addr add 192.168.100.1/24 dev veth0_peer ip netns exec external ip -6 addr add 2003::1/24 dev veth0_pee ip addr add 192.168.100.2/24 dev veth0 ip -6 addr add 2003::2/24 dev veth0 ip route add 172.16.1.0/24 via 192.168.100.1 ip -6 route add 2002::/64 via 2003::1 ip netns exec external ip route add default via 172.16.1.1 ip netns exec external ip -6 route add default via 2002::1 ip netns exec external sysctl net.ipv4.ip_forward=1 ip netns exec external sysctl net.ipv6.ip_forward=1 uuid=$(ovn-nbctl create bfd logical_port=router1-public dst_ip=172.16.1.50 min_tx=250 min_rx=250 detect_mult=10) uuid1=$(ovn-nbctl create bfd logical_port=router1-public dst_ip="2002\:\:50" min_tx=250 min_rx=250 detect_mult=10) uuid2=$(ovn-nbctl create bfd logical_port=router1-public dst_ip="2002\:\:51" min_tx=250 min_rx=250 detect_mult=10) uuid3=$(ovn-nbctl create bfd logical_port=router1-public dst_ip=172.16.1.51 min_tx=250 min_rx=250 detect_mult=10) ovn-nbctl --ecmp lr-route-add router1 192.168.100.0/24 172.16.1.50 ovn-nbctl --bfd=$uuid3 --ecmp lr-route-add router1 192.168.100.0/24 172.16.1.51 ovn-nbctl --ecmp lr-route-add router1 2003::/24 2002::50 ovn-nbctl --bfd=$uuid2 --ecmp lr-route-add router1 2003::/24 2002::51 uid=$(ovn-nbctl --bare --columns _uuid find logical_router_static_route nexthop=172.16.1.50) uid1=$(ovn-nbctl --bare --columns _uuid find logical_router_static_route nexthop="2002\:\:50") ovn-nbctl set logical_router_static_route $route_uuid bfd=$uuid ovn-nbctl set logical_router_static_route $route_uuid1 bfd=$uuid1 ovn-nbctl --wait=hv sync ip netns exec external bfdd-beacon --listen=0.0.0.0 ip netns exec external bfdd-control allow 172.16.1.1 sleep 5 # ovn-nbctl list bfd _uuid : 5e0ce2bc-d556-4e35-b860-7b44e7c61f47 detect_mult : 10 dst_ip : "172.16.1.51" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : up ----------------------------------nexthop is up _uuid : 5df64215-4ef1-4e08-b2b9-82b9ec5b987c detect_mult : 10 dst_ip : "2002::51" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down _uuid : c2b026d3-3f86-4fea-841d-a2361f0651f4 detect_mult : 10 dst_ip : "172.16.1.50" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : up -------------------------------------nexthop is up _uuid : 9ac508df-cca7-4fd9-8638-30388236293f detect_mult : 10 dst_ip : "2002::50" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down ping from vm1 to pc: # ip netns exec vm1 ping 192.168.100.2 PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data. 64 bytes from 192.168.100.2: icmp_seq=1 ttl=62 time=1.24 ms 64 bytes from 192.168.100.2: icmp_seq=2 ttl=62 time=0.074 ms ^C --- 192.168.100.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.074/0.657/1.241/0.584 ms # ovn-sbctl dump-flows router1 | grep '192.168.100.0' table=10(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 192.168.100.0/24), action=(ip.ttl--; flags.loopback = 1; reg8[0..15] = 1; reg8[16..31] = select(1, 2);) now, set one nexthop(172.16.1.50) down: # ovn-sbctl dump-flows router1 | grep '192.168.100.0' table=10(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 192.168.100.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 172.16.1.51; reg1 = 172.16.1.1; eth.src = 40:44:00:00:00:06; outport = "router1-public"; flags.loopback = 1; next;) # ovn-nbctl list bfd _uuid : 5e0ce2bc-d556-4e35-b860-7b44e7c61f47 detect_mult : 10 dst_ip : "172.16.1.51" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : up _uuid : 5df64215-4ef1-4e08-b2b9-82b9ec5b987c detect_mult : 10 dst_ip : "2002::51" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down _uuid : c2b026d3-3f86-4fea-841d-a2361f0651f4 detect_mult : 10 dst_ip : "172.16.1.50" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down --------------------------------nexthop down now _uuid : 9ac508df-cca7-4fd9-8638-30388236293f detect_mult : 10 dst_ip : "2002::50" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down # ip netns exec vm1 ping 192.168.100.2 PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data. 64 bytes from 192.168.100.2: icmp_seq=1 ttl=62 time=1.05 ms 64 bytes from 192.168.100.2: icmp_seq=2 ttl=62 time=0.080 ms 64 bytes from 192.168.100.2: icmp_seq=3 ttl=62 time=0.067 ms ^C --- 192.168.100.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2022ms rtt min/avg/max/mdev = 0.067/0.399/1.052/0.461 ms now make all nexthop bfd down(stop bfd listen,but link is ok) # ip netns exec external bfdd-control stop 172.16.1.1 stopping # ovn-nbctl list bfd _uuid : 5e0ce2bc-d556-4e35-b860-7b44e7c61f47 detect_mult : 10 dst_ip : "172.16.1.51" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down -------------------all down _uuid : 5df64215-4ef1-4e08-b2b9-82b9ec5b987c detect_mult : 10 dst_ip : "2002::51" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down _uuid : c2b026d3-3f86-4fea-841d-a2361f0651f4 detect_mult : 10 dst_ip : "172.16.1.50" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down -----------------------------all down _uuid : 9ac508df-cca7-4fd9-8638-30388236293f detect_mult : 10 dst_ip : "2002::50" external_ids : {} logical_port : router1-public min_rx : 250 min_tx : 250 options : {} status : down # ovn-sbctl dump-flows router1 | grep '192.168.100.0' -------------------------no flows about 192.168.100.0 [root@dell-per740-53 load_balance]# ping fail at this time # ip netns exec vm1 ping 192.168.100.2 PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data. ^C --- 192.168.100.2 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2069ms
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0836