Bug 2007549

Summary: BFD sessions are not created correctly from the parameters of lr-route-add
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Surya Seetharaman <surya>
Component: OVNAssignee: lorenzo bianconi <lorenzo.bianconi>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: FDP 21.CCC: ctrautma, jiji, kfida, lorenzo.bianconi, murali
Target Milestone: ---   
Target Release: FDP 21.I   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn2.13-20.12.0-179.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-09 15:37:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 2007443    

Description Surya Seetharaman 2021-09-24 08:38:48 UTC
Description of problem:

Let's say we do two route add commands:

/usr/bin/ovn-nbctl --timeout=15 --may-exist --bfd --policy=src-ip --ecmp-symmetric-reply lr-route-add GR_ovn-worker 10.244.0.5/32 172.18.0.4 rtoe-GR_ovn-worker
/usr/bin/ovn-nbctl --timeout=15 --may-exist --bfd --policy=src-ip --ecmp-symmetric-reply lr-route-add GR_ovn-worker2 10.244.2.5/32 172.18.0.4 rtoe-GR_ovn-worker2

We'd expect two sessions: one from 0.5 to 0.4 and another from 2.5 to 0.4. Currently the way OVN creates a bfd session is:

Introduce the --bfd option to lr-route-add command.
If the BFD session UUID is provided, it will be used for the OVN route
otherwise the next-hop will be used to perform a lookup in the OVN BFD
table.
If the lookup fails and outport is specified, a new entry in the BFD table
will be created using the nexthop as dst_ip and outport as logical_port.



How reproducible: Always


Steps to Reproduce:
1. Create two routes one from x1 to y1 and another from x2 to y1.
2.
3.

Actual results:
Only one session from x1 to y1 will be created. x2 to y1 will fail because OVN checks that y1 already has a bfd session and won't create a new one.

Expected results:
We should have two bfd sessions.

Comment 1 lorenzo bianconi 2021-09-24 10:15:14 UTC
(In reply to Surya Seetharaman from comment #0)
> Description of problem:
> 
> Let's say we do two route add commands:
> 
> /usr/bin/ovn-nbctl --timeout=15 --may-exist --bfd --policy=src-ip
> --ecmp-symmetric-reply lr-route-add GR_ovn-worker 10.244.0.5/32 172.18.0.4
> rtoe-GR_ovn-worker
> /usr/bin/ovn-nbctl --timeout=15 --may-exist --bfd --policy=src-ip
> --ecmp-symmetric-reply lr-route-add GR_ovn-worker2 10.244.2.5/32 172.18.0.4
> rtoe-GR_ovn-worker2

we need to consider output interface as well looking for an existing bfd session.
For the time being we can create 2 different bfd seesions with different outports and then link them to the static routes with --bfd uuid

> 
> We'd expect two sessions: one from 0.5 to 0.4 and another from 2.5 to 0.4.
> Currently the way OVN creates a bfd session is:
> 
> Introduce the --bfd option to lr-route-add command.
> If the BFD session UUID is provided, it will be used for the OVN route
> otherwise the next-hop will be used to perform a lookup in the OVN BFD
> table.
> If the lookup fails and outport is specified, a new entry in the BFD table
> will be created using the nexthop as dst_ip and outport as logical_port.
> 
> 
> 
> How reproducible: Always
> 
> 
> Steps to Reproduce:
> 1. Create two routes one from x1 to y1 and another from x2 to y1.
> 2.
> 3.
> 
> Actual results:
> Only one session from x1 to y1 will be created. x2 to y1 will fail because
> OVN checks that y1 already has a bfd session and won't create a new one.
> 
> Expected results:
> We should have two bfd sessions.

Comment 3 Surya Seetharaman 2021-09-29 11:01:22 UTC
Tested this PR on an OCP OVN-K cluster and it works as expected. I created two pods on different nodes having the same nexthop and OVN created two BFD sessions one from each of the GW routers on those nodes towards the nexthop/dst_ip.

=======

sh-4.4# ovn-nbctl lr-route-list GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-b-vshc4
IPv4 Routes
              10.128.8.12                10.0.128.4 src-ip rtoe-GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-b-vshc4 ecmp-symmetric-reply bfd
            10.128.0.0/16                100.64.0.1 dst-ip
                0.0.0.0/0                10.0.128.1 dst-ip rtoe-GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-b-vshc4
sh-4.4# ovn-nbctl lr-route-list GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-c-rrq85
IPv4 Routes
             10.128.10.13                10.0.128.4 src-ip rtoe-GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-c-rrq85 ecmp-symmetric-reply bfd
            10.128.0.0/16                100.64.0.1 dst-ip
                0.0.0.0/0                10.0.128.1 dst-ip rtoe-GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-c-rrq85
sh-4.4# ovn-nbctl list bfd
_uuid               : a646da00-f201-4048-923b-82bac426c91a
detect_mult         : []
dst_ip              : "10.0.128.4"
external_ids        : {}
logical_port        : rtoe-GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-b-vshc4
min_rx              : []
min_tx              : []
options             : {}
status              : down

_uuid               : 00c14ced-6c0c-4b9e-8772-4eff9411bad6
detect_mult         : []
dst_ip              : "10.0.128.4"
external_ids        : {}
logical_port        : rtoe-GR_ci-ln-xwgizmt-f76d1-f8gv9-worker-c-rrq85
min_rx              : []
min_tx              : []
options             : {}
status              : down

Would be good if we could move this patch along.

Comment 4 Jianlin Shi 2021-10-19 02:05:27 UTC
tested with following script:

systemctl start openvswitch
systemctl start ovn-northd                                   
ovn-nbctl set-connection ptcp:6641          
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.40.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.40.25
systemctl restart ovn-controller         
                                                   
ovn-nbctl lr-add r1
ovn-nbctl lrp-add r1 r1-net1 00:00:00:01:ff:01 192.168.100.1/24
                                                           
ovn-nbctl lr-add r2                                        
ovn-nbctl lrp-add r2 r2-net2 00:00:00:02:ff:01 192.168.100.1/24
                                                        
ovn-nbctl lr-add r3
ovn-nbctl lrp-add r3 r3-net3 00:00:00:03:ff:01 192.168.200.1/24
                                                                          
ovn-nbctl ls-add pub
ovn-nbctl lrp-add r1 r1-pub 00:00:00:11:ff:01 172.18.1.124/24
ovn-nbctl lrp-set-gateway-chassis r1-pub hv1      
ovn-nbctl lsp-add pub pub-r1      
ovn-nbctl lsp-set-type pub-r1 router  
ovn-nbctl lsp-set-addresses pub-r1 router
ovn-nbctl lsp-set-options pub-r1 router-port=r1-pub                    
                 
ovn-nbctl lrp-add r2 r2-pub 00:00:00:12:ff:01 172.18.2.110/24
ovn-nbctl lrp-set-gateway-chassis r2-pub hv1
ovn-nbctl lsp-add pub pub-r2                         
ovn-nbctl lsp-set-type pub-r2 router                 
ovn-nbctl lsp-set-addresses pub-r2 router                 
ovn-nbctl lsp-set-options pub-r2 router-port=r2-pub        

ovn-nbctl lrp-add r3 r3-pub 00:00:00:13:ff:01 172.18.1.173/24
ovn-nbctl lrp-set-gateway-chassis r3-pub hv1
ovn-nbctl lsp-add pub pub-r3
ovn-nbctl lsp-set-type pub-r3 router
ovn-nbctl lsp-set-addresses pub-r3 router
ovn-nbctl lsp-set-options pub-r3 router-port=r3-pub


ovn-nbctl --bfd lr-route-add r1 0.0.0.0/0 172.18.1.1 r1-pub
ovn-nbctl --bfd lr-route-add r3 0.0.0.0/0 172.18.1.1 r3-pub
#ovn-nbctl lr-route-add r2 172.18.2.10 172.18.2.1 r2-pub
#ovn-nbctl lr-route-add r2 172.18.2.12 172.18.2.1 r2-pub

ovs-vsctl add-br br-provider
ovs-vsctl set open . external-ids:ovn-bridge-mappings=provider:br-provider

ovn-nbctl lsp-add pub ln
ovn-nbctl lsp-set-options ln network_name=provider
ovn-nbctl lsp-set-type ln localnet
ovn-nbctl lsp-set-addresses ln unknown

ovs-vsctl add-port br-provider ext1 -- set interface ext1 type=internal
ip netns add ext1
ip link set ext1 netns ext1
ip netns exec ext1 ip link set ext1 up
ip netns exec ext1 ip addr add 172.18.1.1/24 dev ext1
ip netns exec ext1 ip addr add 172.18.2.1/24 dev ext1
ip netns exec ext1 sysctl -w net.ipv4.conf.all.rp_filter=0
ip netns exec ext1 sysctl -w net.ipv4.conf.all.forwarding=1

ovn-nbctl list bfd 
ovn-nbctl lr-route-list r1
ovn-nbctl lr-route-list r3

reproduced on ovn-2021-21.06.0-24:

+ ovn-nbctl list bfd
_uuid               : 62852b59-9235-4195-8886-ee607c80c0fd                                            
detect_mult         : []
dst_ip              : "172.18.1.1"                                                                    
external_ids        : {}
logical_port        : r1-pub                                                                          
min_rx              : []                                                                              
min_tx              : []                                                                              
options             : {}
status              : down
+ ovn-nbctl lr-route-list r1                                                                          
IPv4 Routes
                0.0.0.0/0                172.18.1.1 dst-ip r1-pub bfd                                 
+ ovn-nbctl lr-route-list r3                                                                          
IPv4 Routes
                0.0.0.0/0                172.18.1.1 dst-ip r3-pub bfd

Verified on ovn-2021-21.09.0-12:

[root@dell-per740-12 bz2007549]# rpm -qa | grep -E "openvswitch2.16|ovn-2021"
ovn-2021-21.09.0-12.el8fdp.x86_64
ovn-2021-central-21.09.0-12.el8fdp.x86_64
ovn-2021-host-21.09.0-12.el8fdp.x86_64
python3-openvswitch2.16-2.16.0-16.el8fdp.x86_64
openvswitch2.16-2.16.0-16.el8fdp.x86_64
openvswitch2.16-test-2.16.0-16.el8fdp.noarch

+ ovn-nbctl list bfd                                                                                  
_uuid               : 486b6b5a-81eb-4d5e-935f-c64ec0496052                                                                                                         
detect_mult         : []                                                                              
dst_ip              : "172.18.1.1"                                                                    
external_ids        : {}
logical_port        : r3-pub                                   
min_rx              : []                                   
min_tx              : []                                                                              
options             : {}                                                                              
status              : down                              
                   
_uuid               : 53c65703-25a6-43e4-a29a-eb1d0591148f     
detect_mult         : []                                                                              
dst_ip              : "172.18.1.1"
external_ids        : {}                                     
logical_port        : r1-pub                                                                          
min_rx              : []                                                                              
min_tx              : []                                                                              
options             : {}                 
status              : down                                             
+ ovn-nbctl lr-route-list r1
IPv4 Routes                                                                                           
                0.0.0.0/0                172.18.1.1 dst-ip r1-pub bfd
+ ovn-nbctl lr-route-list r3                                                                          
IPv4 Routes                                          
                0.0.0.0/0                172.18.1.1 dst-ip r3-pub bfd

Comment 5 Jianlin Shi 2021-10-19 02:07:13 UTC
also verified on ovn2.13-20.12.0-185.el8:

[root@dell-per740-12 bz2007549]# rpm -qa | grep -E "openvswitch2.16|ovn2.13"
ovn2.13-central-20.12.0-185.el8fdp.x86_64                                                             
ovn2.13-host-20.12.0-185.el8fdp.x86_64                                                                
python3-openvswitch2.16-2.16.0-16.el8fdp.x86_64                                                       
ovn2.13-20.12.0-185.el8fdp.x86_64                                                                     
openvswitch2.16-2.16.0-16.el8fdp.x86_64                                                               
openvswitch2.16-test-2.16.0-16.el8fdp.noarch

+ ovn-nbctl list bfd                                                                                  
_uuid               : 4291183c-1baa-4b5d-9d6b-0913846cb516
detect_mult         : []                                                                              
dst_ip              : "172.18.1.1"                                                                    
external_ids        : {}                                                                              
logical_port        : r1-pub                                                                          
min_rx              : []                                                                              
min_tx              : []                                                                              
options             : {}                                                                              
status              : down                                                                            
                                                                                                      
_uuid               : b953e8ba-e4d9-4e64-af5c-32ad32e8e245
detect_mult         : []                                                                              
dst_ip              : "172.18.1.1"                                                                    
external_ids        : {}                                                                              
logical_port        : r3-pub                                                                          
min_rx              : []                                                                              
min_tx              : []                                                                              
options             : {}                                                                              
status              : down                                                                            
+ ovn-nbctl lr-route-list r1                                                                          
IPv4 Routes                                                                                           
                0.0.0.0/0                172.18.1.1 dst-ip r1-pub bfd
+ ovn-nbctl lr-route-list r3                                                                          
IPv4 Routes                                                                                           
                0.0.0.0/0                172.18.1.1 dst-ip r3-pub bfd

Comment 8 Jianlin Shi 2021-10-20 01:37:23 UTC
set VERIFIED per comment 4

Comment 10 errata-xmlrpc 2021-12-09 15:37:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:5059