Description of problem: [OVN]load balancer in logical_router doesn;t work with nat Version-Release number of selected component (if applicable): # rpm -qa|grep ovn ovn-2021-central-21.12.0-11.el8fdp.x86_64 ovn-2021-host-21.12.0-11.el8fdp.x86_64 ovn-2021-21.12.0-11.el8fdp.x86_64 How reproducible: always Steps to Reproduce: ovn-nbctl ls-add public uuid1=`ovn-nbctl create load_balancer vips:30.0.0.1="172.16.0.103,172.16.0.102"` uuid2=`ovn-nbctl create load_balancer vips:30.0.0.2="172.16.0.103,172.16.0.101"` uuid3=`ovn-nbctl create load_balancer vips:30.0.0.3="172.16.0.102,172.16.0.101"` uuid4=`ovn-nbctl create load_balancer vips:30.0.0.4="172.16.0.103,172.16.0.102"` lbg=$(ovn-nbctl create load_balancer_group name=lbg -- \ add load_balancer_group lbg load_balancer $uuid1 -- \ add load_balancer_group lbg load_balancer $uuid2 -- \ add load_balancer_group lbg load_balancer $uuid3 -- \ add load_balancer_group lbg load_balancer $uuid4) ovn-nbctl --wait=sb add logical_switch public load_balancer_group $lbg # r1 i=1 for m in `seq 0 9`;do for n in `seq 1 99`;do ovn-nbctl lr-add r${i} ovn-nbctl lrp-add r${i} r${i}_public 00:de:ad:ff:$m:$n 172.16.$m.$n/16 ovn-nbctl lrp-add r${i} r${i}_s${i} 00:de:ad:fe:$m:$n 173.$m.$n.1/24 ovn-nbctl lr-nat-add r${i} dnat_and_snat 172.16.${m}.$((n+100)) 173.$m.$n.2 ovn-nbctl lrp-set-gateway-chassis r${i}_public hv1 ovn-nbctl --wait=sb add logical_router r${i} load_balancer_group $lbg # s1 ovn-nbctl ls-add s${i} # s1 - r1 ovn-nbctl lsp-add s${i} s${i}_r${i} ovn-nbctl lsp-set-type s${i}_r${i} router ovn-nbctl lsp-set-addresses s${i}_r${i} "00:de:ad:fe:$m:$n 173.$m.$n.1" ovn-nbctl lsp-set-options s${i}_r${i} router-port=r${i}_s${i} # s1 - vm1 ovn-nbctl lsp-add s$i vm$i ovn-nbctl lsp-set-addresses vm$i "00:de:ad:01:$m:$n 173.$m.$n.2" ovn-nbctl lrp-add r$i r${i}_public 40:44:00:00:$m:$n 172.16.$m.$n/16 ovn-nbctl lsp-add public public_r${i} ovn-nbctl lsp-set-type public_r${i} router ovn-nbctl lsp-set-addresses public_r${i} router ovn-nbctl lsp-set-options public_r${i} router-port=r${i}_public nat-addresses=router let i++ if [ $i -gt $round ];then break; fi done if [ $i -gt $round ];then break; fi done done ovn-nbctl lsp-add public ln_p1 ovn-nbctl lsp-set-addresses ln_p1 unknown ovn-nbctl lsp-set-type ln_p1 localnet ovn-nbctl lsp-set-options ln_p1 network_name=nattest #add host vm1 ip netns add vm1 ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal ip link set vm1 netns vm1 ip netns exec vm1 ip link set vm1 address 00:de:ad:01:00:01 ip netns exec vm1 ip addr add 173.0.1.2/24 dev vm1 ip netns exec vm1 ip link set vm1 up ovs-vsctl set Interface vm1 external_ids:iface-id=vm1 ip netns add vm2 ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal ip link set vm2 netns vm2 ip netns exec vm2 ip link set vm2 address 00:de:ad:01:00:02 ip netns exec vm2 ip addr add 173.0.2.2/24 dev vm2 ip netns exec vm2 ip link set vm2 up ovs-vsctl set Interface vm2 external_ids:iface-id=vm2 ip netns add vm3 ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal ip link set vm3 netns vm3 ip netns exec vm3 ip link set vm3 address 00:de:ad:01:00:03 ip netns exec vm3 ip addr add 173.0.3.2/24 dev vm3 ip netns exec vm3 ip link set vm3 up ovs-vsctl set Interface vm3 external_ids:iface-id=vm3 #set provide network ovs-vsctl add-br nat_test ip link set nat_test up ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=nattest:nat_test ip netns add vm0 ip link set vm0 netns vm0 ip netns exec vm0 ip link set vm0 address 00:00:00:00:00:01 ip netns exec vm0 ip addr add 172.16.0.100/16 dev vm0 ip netns exec vm0 ip link set vm0 up ovs-vsctl add-port nat_test vm0 -- set interface vm0 type=internal ip link set vm0 netns vm0 ip netns exec vm0 ip link set vm0 address 00:00:00:00:00:01 ip netns exec vm0 ip addr add 172.16.0.100/16 dev vm0 ip netns exec vm0 ip link set vm0 up ovs-vsctl set Interface vm0 external_ids:iface-id=vm0 ip netns exec vm1 ip route add default via 173.0.1.1 ip netns exec vm2 ip route add default via 173.0.2.1 ip netns exec vm3 ip route add default via 173.0.3.1 ovn-nbctl lr-nat-del r1 dnat_and_snat 172.16.0.101 ovn-nbctl lr-nat-add r1 dnat_and_snat 172.16.0.101 173.0.1.2 vm1 00:00:00:01:02:03 then ping the vip: ip netns exec vm1 ping 30.0.0.1 -c 5' PING 30.0.0.1 (30.0.0.1) 56(84) bytes of data. --- 30.0.0.1 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4084ms ip netns exec vm2 ping 30.0.0.2 -c 5' PING 30.0.0.2 (30.0.0.2) 56(84) bytes of data. --- 30.0.0.2 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4078ms ip netns exec vm3 ping 30.0.0.3 -c 5' PING 30.0.0.3 (30.0.0.3) 56(84) bytes of data. --- 30.0.0.3 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4136ms ip netns exec vm3 ping 30.0.0.1 -c 5' PING 30.0.0.1 (30.0.0.1) 56(84) bytes of data. --- 30.0.0.1 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4093ms ip netns exec vm2 ping 30.0.0.4 -c 5' PING 30.0.0.4 (30.0.0.4) 56(84) bytes of data. 64 bytes from 172.16.0.102: icmp_seq=1 ttl=62 time=1.91 ms -----this looks good,but the return ip should be 30.0.0.4 64 bytes from 172.16.0.102: icmp_seq=2 ttl=62 time=0.550 ms 64 bytes from 172.16.0.102: icmp_seq=3 ttl=62 time=0.064 ms 64 bytes from 172.16.0.102: icmp_seq=4 ttl=62 time=0.068 ms 64 bytes from 172.16.0.102: icmp_seq=5 ttl=62 time=0.057 ms --- 30.0.0.4 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4076ms Actual results: ping fail and return ip wrong. Expected results: ping pass and the return ip should be the vip. Additional info:
it is not nessesary to use load_balancer_group. I change the LBG to LB, it has the same issue. ovn-nbctl ls-add public uuid1=`ovn-nbctl create load_balancer vips:30.0.0.1="172.16.0.103,172.16.0.102"` # r1 i=1 for m in `seq 0 9`;do for n in `seq 1 99`;do ovn-nbctl lr-add r${i} ovn-nbctl lrp-add r${i} r${i}_public 00:de:ad:ff:$m:$n 172.16.$m.$n/16 ovn-nbctl lrp-add r${i} r${i}_s${i} 00:de:ad:fe:$m:$n 173.$m.$n.1/24 ovn-nbctl lr-nat-add r${i} dnat_and_snat 172.16.${m}.$((n+100)) 173.$m.$n.2 ovn-nbctl lrp-set-gateway-chassis r${i}_public hv1 ovn-nbctl --wait=sb add logical_router r${i} load_balancer $uuid1 # s1 ovn-nbctl ls-add s${i} # s1 - r1 ovn-nbctl lsp-add s${i} s${i}_r${i} ovn-nbctl lsp-set-type s${i}_r${i} router ovn-nbctl lsp-set-addresses s${i}_r${i} "00:de:ad:fe:$m:$n 173.$m.$n.1" ovn-nbctl lsp-set-options s${i}_r${i} router-port=r${i}_s${i} # s1 - vm1 ovn-nbctl lsp-add s$i vm$i ovn-nbctl lsp-set-addresses vm$i "00:de:ad:01:$m:$n 173.$m.$n.2" ovn-nbctl lrp-add r$i r${i}_public 40:44:00:00:$m:$n 172.16.$m.$n/16 ovn-nbctl lsp-add public public_r${i} ovn-nbctl lsp-set-type public_r${i} router ovn-nbctl lsp-set-addresses public_r${i} router ovn-nbctl lsp-set-options public_r${i} router-port=r${i}_public nat-addresses=router let i++ if [ $i -gt $round ];then break; fi done if [ $i -gt $round ];then break; fi done done ovn-nbctl lsp-add public ln_p1 ovn-nbctl lsp-set-addresses ln_p1 unknown ovn-nbctl lsp-set-type ln_p1 localnet ovn-nbctl lsp-set-options ln_p1 network_name=nattest #add host vm1 ip netns add vm1 ovs-vsctl add-port br-int vm1 -- set interface vm1 type=internal ip link set vm1 netns vm1 ip netns exec vm1 ip link set vm1 address 00:de:ad:01:00:01 ip netns exec vm1 ip addr add 173.0.1.2/24 dev vm1 ip netns exec vm1 ip link set vm1 up ovs-vsctl set Interface vm1 external_ids:iface-id=vm1 ip netns add vm2 ovs-vsctl add-port br-int vm2 -- set interface vm2 type=internal ip link set vm2 netns vm2 ip netns exec vm2 ip link set vm2 address 00:de:ad:01:00:02 ip netns exec vm2 ip addr add 173.0.2.2/24 dev vm2 ip netns exec vm2 ip link set vm2 up ovs-vsctl set Interface vm2 external_ids:iface-id=vm2 ip netns add vm3 ovs-vsctl add-port br-int vm3 -- set interface vm3 type=internal ip link set vm3 netns vm3 ip netns exec vm3 ip link set vm3 address 00:de:ad:01:00:03 ip netns exec vm3 ip addr add 173.0.3.2/24 dev vm3 ip netns exec vm3 ip link set vm3 up ovs-vsctl set Interface vm3 external_ids:iface-id=vm3 #set provide network ovs-vsctl add-br nat_test ip link set nat_test up ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-mappings=nattest:nat_test ip netns add vm0 ip link set vm0 netns vm0 ip netns exec vm0 ip link set vm0 address 00:00:00:00:00:01 ip netns exec vm0 ip addr add 172.16.0.100/16 dev vm0 ip netns exec vm0 ip link set vm0 up ovs-vsctl add-port nat_test vm0 -- set interface vm0 type=internal ip link set vm0 netns vm0 ip netns exec vm0 ip link set vm0 address 00:00:00:00:00:01 ip netns exec vm0 ip addr add 172.16.0.100/16 dev vm0 ip netns exec vm0 ip link set vm0 up ovs-vsctl set Interface vm0 external_ids:iface-id=vm0 ip netns exec vm1 ip route add default via 173.0.1.1 ip netns exec vm2 ip route add default via 173.0.2.1 ip netns exec vm3 ip route add default via 173.0.3.1 ovn-nbctl lr-nat-del r1 dnat_and_snat 172.16.0.101 ovn-nbctl lr-nat-add r1 dnat_and_snat 172.16.0.101 173.0.1.2 vm1 00:00:00:01:02:03 then ,ping from vm1 to th vip: ip netns exec vm1 ping 30.0.0.1 PING 30.0.0.1 (30.0.0.1) 56(84) bytes of data. ^C --- 30.0.0.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2072ms
We can't seem to reproduce this outside of OSP, specifically we're not sure if the non-working state is hitting the bug or we just misconfigured things. I have some questions: 1) I assume this is a regression, can you please provide a working OVN version? With working version we could make sure our environment configuration works with the version and doesn't with latest. 2) Do we need to create 1000 ports like in the example? We tried to come up with a simplest possible solution that would show the bug. 3) Do we need the provider net_test bridge? 4) Is ICMP supposed to work with load balancers? Thanks for your help
Also from networking perspective, the 30.0.0.1 IP will use the default route - do we expect the router to notice it's a VIP and pick that up even though the router is not connected to any switch with that IP?
(In reply to Jakub Libosvar from comment #2) > We can't seem to reproduce this outside of OSP, specifically we're not sure > if the non-working state is hitting the bug or we just misconfigured things. > > I have some questions: > > 1) I assume this is a regression, can you please provide a working OVN > version? With working version we could make sure our environment > configuration works with the version and doesn't with latest. > this version is working.ovn-2021/21.09.1/23.el8fdp > 2) Do we need to create 1000 ports like in the example? We tried to come up > with a simplest possible solution that would show the bug. > no, we only need 2 ls and 2 lr. in my env,I have 3 ls and 3 lr. > 3) Do we need the provider net_test bridge? I think it is not nessesary. > > 4) Is ICMP supposed to work with load balancers? I thought it is. > > Thanks for your help
> 4) Is ICMP supposed to work with load balancers? If a load balancer VIP has no port specified, then all traffic types are permitted, including ICMP
I used the instructions from https://bugzilla.redhat.com/show_bug.cgi?id=2044127#c1 to set up this scenario, and I found that pings were failing. VM1 sends a ping out, it traverses logical switch s1 and reaches logical router r1. At this point, the load balancer translates the 30.0.0.1 destination into one of the load balancer backends. For me, it translated to 172.16.0.103. The packet then exits r1 and hits the public switch. From here, it goes into router r3, where the address is DNATted to 173.0.3.2 and reaches vm3. At this point, the packet has essentially been DNATted twice, but has not undergone any SNAT. vm3 therefore attempts to respond by sending its response to 173.0.1.2 (vm1's IP address). Since logical router r3 has no route to that address, the ping response packet is dropped by r3. As a workaround you can add a static route to r3, then you can route the ping back to vm1. ovn-nbctl lr-route-add r3 173.0.1.0/24 172.16.0.1 I think the issue here is that the ping request packet that exits r1 is not SNATted. If it were, then the source IP address would become 172.16.0.101 when the packet exits r1. If vm3 were to attempt to respond to 172.16.0.101, then the packet should get routed back to vm1. I will investigate and comment when I have a solution.
I did some investigation, and it appears that this scenario worked up until commit 4deac4509abbedd6ffaecf27eed01ddefccea40a (northd: Use ct_(snat/dnat)_in_czone action for distributed routers.) Non-working conntrack: [root@ovn-chassis-1 /]# ovs-appctl dpctl/dump-conntrack icmp,orig=(src=173.0.1.2,dst=30.0.0.1,id=344,type=8,code=0),reply=(src=172.16.0.103,dst=173.0.1.2,id=344,type=0,code=0),zone=1,labels=0x2 icmp,orig=(src=173.0.1.2,dst=172.16.0.103,id=344,type=8,code=0),reply=(src=173.0.3.2,dst=173.0.1.2,id=344,type=0,code=0),zone=11 Working conntrack: [root@ovn-chassis-1 /]# ovs-appctl dpctl/dump-conntrack icmp,orig=(src=173.0.1.2,dst=30.0.0.1,id=310,type=8,code=0),reply=(src=172.16.0.103,dst=173.0.1.2,id=310,type=0,code=0),zone=1,labels=0x2 icmp,orig=(src=172.16.0.101,dst=172.16.0.103,id=310,type=8,code=0),reply=(src=173.0.3.2,dst=172.16.0.101,id=310,type=0,code=0),zone=12 icmp,orig=(src=173.0.1.2,dst=172.16.0.103,id=310,type=8,code=0),reply=(src=172.16.0.103,dst=172.16.0.101,id=310,type=0,code=0),zone=2 In the working version, SNAT and DNAT are performed in separate CT zones (zone 1 is the DNAT and zone 2 is the SNAT). In the non-working version, I think that the SNAT and DNAT are supposed to both be done in the same zone (zone 1) but apparently only the DNAT is done.
Review posted here: https://patchwork.ozlabs.org/project/ovn/patch/20220309193810.246221-1-mmichels@redhat.com/
Just one observation here. I'm sure the issue will not be hit if the load balancer is associated to the logical switch of the source port. Since OSP associates a load balancer to all router and to all logical switches connected to it (except the provider network logical switch), the issue is not seen there.
Moving this to MODIFIED. This was actually merged into OVN in late September, but I forgot to cite the BZ in the commit message. Therefore, the issue did not automatically get updated. This was fixed in 22.09, 22.06, and 22.03.
(In reply to Mark Michelson from comment #10) > Moving this to MODIFIED. This was actually merged into OVN in late > September, but I forgot to cite the BZ in the commit message. Therefore, the > issue did not automatically get updated. This was fixed in 22.09, 22.06, and > 22.03. It seems not fix in version rhel9 for 22.09, 22.06 and 22.03 it is ok for rhel8.
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.