Description of problem: We see with ovnkube when using symmetric reply for for return ingress traffic to go to the originating gateway, that it does not work for ipv6 but does work for ipv4.
I believe Mark reproduced this with his AT test with OVN.
We were able to reproduce this in a KIND setup. ovn-trace shows the reply packet should make it, however in dpctl we see: [root@ovn-control-plane ~]# ovs-dpctl dump-flows | grep fd00:10:244:1::b recirc_id(0x61e),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x3),eth(src=0a:58:0a:f4:00:01,dst=0a:58:0a:f4:00:0b),eth_type(0x86dd),ipv6(src=fc00:f853:ccd:e793::3,dst=fd00:10:244:1::b,proto=58,hlimit=62,frag=no), packets:63, bytes:7434, used :0.936s, actions:10 recirc_id(0x38f),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),ct_label(0),eth(src=02:42:ac:12:00:03,dst=02:42:ac:12:00:02),eth_type(0x86dd),ipv6(src=fc00:f853:ccd:e793::/ffff:ffff:ffff:ffff::,dst=fd00:10:244:1::b,proto=58,hlimit=64,frag=no), packets:64, bytes:7552, used:0.936s, actions:ct_clear,ct(zone=15),recirc(0x61c) recirc_id(0x61d),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0x200000000000000000000/0xffff00000000000000000003),eth(src=02:42:ac:12:00:03,dst=02:42:ac:12:00:02),eth_type(0x86dd),ipv6(src=fc00:f853:ccd:e793::3,dst=fd00:10:244:1::b,proto=58,hli mit=64,frag=no), packets:63, bytes:7434, used:0.936s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:0a:f4:00:01,dst=0a:58:0a:f4:00:0b)),set(ipv6(hlimit=62)),ct(zone=24),recirc(0x61e) recirc_id(0x61f),in_port(10),ct_state(-new+est-rel+rpl-inv+trk),ct_label(0/0xffff00000000000000000003),eth(src=0a:58:0a:f4:00:0b,dst=0a:58:0a:f4:00:01),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,dst=fc00:f853:ccd:e793::3,proto=58,hlimit=64,frag=no), pack ets:64, bytes:7552, used:0.937s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:02)),set(ipv6(hlimit=63)),ct(zone=15),recirc(0x620) recirc_id(0x621),in_port(10),ct_state(-new+est-rel+rpl-inv+trk),ct_label(0/0x1),eth(),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,proto=58,hlimit=63,frag=no), packets:64, bytes:7552, used:0.937s, actions:set(ipv6(hlimit=62)),hash(l4(0)),recirc(0x622) recirc_id(0),in_port(10),eth(src=0a:58:0a:f4:00:0b,dst=0a:58:0a:f4:00:01),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,dst=fc00:f853:ccd:e793::3,proto=58,hlimit=64,frag=no), packets:64, bytes:7552, used:0.937s, actions:ct(zone=24),recirc(0x61f) [root@ovn-control-plane ~]# ovs-dpctl dump-flows | grep 0x61f recirc_id(0x61f),in_port(10),ct_state(-new+est-rel+rpl-inv+trk),ct_label(0/0xffff00000000000000000003),eth(src=0a:58:0a:f4:00:0b,dst=0a:58:0a:f4:00:01),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,dst=fc00:f853:ccd:e793::3,proto=58,hlimit=64,frag=no), pack ets:84, bytes:9912, used:0.471s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:02)),set(ipv6(hlimit=63)),ct(zone=15),recirc(0x620) recirc_id(0),in_port(10),eth(src=0a:58:0a:f4:00:0b,dst=0a:58:0a:f4:00:01),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,dst=fc00:f853:ccd:e793::3,proto=58,hlimit=64,frag=no), packets:84, bytes:9912, used:0.471s, actions:ct(zone=24),recirc(0x61f) [root@ovn-control-plane ~]# ovs-dpctl dump-flows | grep 0x620 recirc_id(0x61f),in_port(10),ct_state(-new+est-rel+rpl-inv+trk),ct_label(0/0xffff00000000000000000003),eth(src=0a:58:0a:f4:00:0b,dst=0a:58:0a:f4:00:01),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,dst=fc00:f853:ccd:e793::3,proto=58,hlimit=64,frag=no), pack ets:91, bytes:10738, used:0.397s, actions:ct_clear,ct_clear,ct_clear,set(eth(src=0a:58:64:40:00:01,dst=0a:58:64:40:00:02)),set(ipv6(hlimit=63)),ct(zone=15),recirc(0x620) recirc_id(0x620),in_port(10),eth(),eth_type(0x86dd),ipv6(frag=no), packets:91, bytes:10738, used:0.397s, actions:ct(zone=15,nat),recirc(0x621) [root@ovn-control-plane ~]# ovs-dpctl dump-flows | grep 0x621 recirc_id(0x621),in_port(10),ct_state(-new+est-rel+rpl-inv+trk),ct_label(0/0x1),eth(),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,proto=58,hlimit=63,frag=no), packets:104, bytes:12272, used:0.654s, actions:set(ipv6(hlimit=62)),hash(l4(0)),recirc(0x622) recirc_id(0x620),in_port(10),eth(),eth_type(0x86dd),ipv6(frag=no), packets:104, bytes:12272, used:0.654s, actions:ct(zone=15,nat),recirc(0x621) [root@ovn-control-plane ~]# ovs-dpctl dump-flows | grep 0x622 recirc_id(0x622),dp_hash(0xb/0xf),in_port(10),ct_state(-new+rpl+trk),ct_label(0x200000000000000000000/0xffff00000000000000000000),eth(src=0a:58:64:40:00:01),eth_type(0x86dd),ipv6(frag=no), packets:120, bytes:14160, used:0.089s, actions:drop <----DROPPED recirc_id(0x621),in_port(10),ct_state(-new+est-rel+rpl-inv+trk),ct_label(0/0x1),eth(),eth_type(0x86dd),ipv6(src=fd00:10:244:1::b,proto=58,hlimit=63,frag=no), packets:120, bytes:14160, used:0.089s, actions:set(ipv6(hlimit=62)),hash(l4(0)),recirc(0x622) In conntrack the SNAT and DNAT entries are never seeing the syn/ack and going established. Will attach the traces.
Created attachment 1782855 [details] ovn, ofproto traces for reply packet
https://patchwork.ozlabs.org/project/ovn/list/?series=244971
v2: https://patchwork.ozlabs.org/project/ovn/list/?series=245131
Patch applied upstream, now being backported to downstream OVN.
test with following script: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.173.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.173.25 systemctl restart ovn-controller # Logical network: # Alice is connected to gateway router R1. R1 is connected to two "external" # routers, R2 and R3 via an "ext" switch. # Bob is connected to both R2 and R3. R1 contains two ECMP routes, one through R2 # and one through R3, to Bob. # # alice -- R1 -- ext ---- R2 # | \ # | bob # | / # + ----- R3 # # For this test, Bob sends request traffic through R2 to Alice. We want to ensure that # all response traffic from Alice is routed through R2 as well. ovn-nbctl create Logical_Router name=R1 options:chassis=hv1 ovn-nbctl create Logical_Router name=R2 ovn-nbctl create Logical_Router name=R3 ovn-nbctl ls-add alice ovn-nbctl ls-add bob ovn-nbctl ls-add ext # connect alice to R1 ovn-nbctl lrp-add R1 alice 00:00:01:01:02:03 fd01::1/64 ovn-nbctl lsp-add alice rp-alice -- set Logical_Switch_Port rp-alice \ type=router options:router-port=alice addresses='"00:00:01:01:02:03"' # connect bob to R2 ovn-nbctl lrp-add R2 R2_bob 00:00:02:01:02:03 fd07::2/64 ovn-nbctl lsp-add bob rp2-bob -- set Logical_Switch_Port rp2-bob \ type=router options:router-port=R2_bob addresses='"00:00:02:01:02:03"' # connect bob to R3 ovn-nbctl lrp-add R3 R3_bob 00:00:02:01:02:04 fd07::3/64 ovn-nbctl lsp-add bob rp3-bob -- set Logical_Switch_Port rp3-bob \ type=router options:router-port=R3_bob addresses='"00:00:02:01:02:04"' # Connect R1 to ext ovn-nbctl lrp-add R1 R1_ext 00:00:04:01:02:03 fd02::1/64 ovn-nbctl lsp-add ext r1-ext -- set Logical_Switch_Port r1-ext \ type=router options:router-port=R1_ext addresses='"00:00:04:01:02:03"' # Connect R2 to ext ovn-nbctl lrp-add R2 R2_ext 00:00:04:01:02:04 fd02::2/64 ovn-nbctl lsp-add ext r2-ext -- set Logical_Switch_Port r2-ext \ type=router options:router-port=R2_ext addresses='"00:00:04:01:02:04"' # Connect R3 to ext ovn-nbctl lrp-add R3 R3_ext 00:00:04:01:02:05 fd02::3/64 ovn-nbctl lsp-add ext r3-ext -- set Logical_Switch_Port r3-ext \ type=router options:router-port=R3_ext addresses='"00:00:04:01:02:05"' # Install ECMP routes for alice. ovn-nbctl --ecmp-symmetric-reply --policy="src-ip" lr-route-add R1 fd01::/126 fd02::2 ovn-nbctl --ecmp-symmetric-reply --policy="src-ip" lr-route-add R1 fd01::/126 fd02::3 ovn-nbctl lr-route-add R2 fd01::/64 fd02::1 ovn-nbctl lr-route-add R3 fd01::/64 fd02::1 ovn-nbctl lsp-add alice alice1 \ -- lsp-set-addresses alice1 "f0:00:00:01:02:04 fd01::2" ovn-nbctl lsp-add bob bob1 \ -- lsp-set-addresses bob1 "f0:00:00:01:02:06 fd07::1" ovs-vsctl add-port br-int alice1 -- set interface alice1 type=internal external_ids:iface-id=alice1 ip netns add alice1 ip link set alice1 netns alice1 ip netns exec alice1 ip link set alice1 address f0:00:00:01:02:04 ip netns exec alice1 ip addr add fd01::2/64 dev alice1 ip netns exec alice1 ip link set alice1 up ip netns exec alice1 ip -6 route add default via fd01::1 ovs-vsctl add-port br-int bob1 -- set interface bob1 type=internal external_ids:iface-id=bob1 ip netns add bob1 ip link set bob1 netns bob1 ip netns exec bob1 ip link set bob1 address f0:00:00:01:02:06 ip netns exec bob1 ip addr add fd07::1/64 dev bob1 ip netns exec bob1 ip link set bob1 up ip netns exec bob1 ip -6 route add default via fd07::2 sleep 2 ovn-nbctl --wait=hv sync ip netns exec bob1 ping6 -c 2 -i 0.3 -w 15 fd01::2 ovs-appctl dpctl/del-flows ip netns exec bob1 ping6 -c 20 -i 0.3 -w 15 fd01::2 reproduced on ovn2.13-20.12.0-135: [root@wsfd-advnetlab18 bz1959008]# rpm -qa | grep -E "openvswitch2.13|ovn2.13" ovn2.13-host-20.12.0-135.el7fdp.x86_64 ovn2.13-central-20.12.0-135.el7fdp.x86_64 openvswitch2.13-2.13.0-98.el7fdp.x86_64 ovn2.13-20.12.0-135.el7fdp.x86_64 + ovn-nbctl --wait=hv sync + ip netns exec bob1 ping6 -c 2 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes --- fd01::2 ping statistics --- 50 packets transmitted, 0 received, 100% packet loss, time 14841ms + ovs-appctl dpctl/del-flows + ip netns exec bob1 ping6 -c 20 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes --- fd01::2 ping statistics --- 50 packets transmitted, 0 received, 100% packet loss, time 14839ms <=== ping failed Verified on ovn2.13-20.12.0-140: [root@wsfd-advnetlab18 bz1959008]# rpm -qa | grep -E "openvswitch2.13|ovn2.13" ovn2.13-20.12.0-140.el7fdp.x86_64 ovn2.13-host-20.12.0-140.el7fdp.x86_64 openvswitch2.13-2.13.0-98.el7fdp.x86_64 ovn2.13-central-20.12.0-140.el7fdp.x86_64 + ovn-nbctl --wait=hv sync + ip netns exec bob1 ping6 -c 2 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes 64 bytes from fd01::2: icmp_seq=1 ttl=62 time=4.83 ms 64 bytes from fd01::2: icmp_seq=2 ttl=62 time=1.46 ms --- fd01::2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 301ms rtt min/avg/max/mdev = 1.460/3.145/4.831/1.686 ms + ovs-appctl dpctl/del-flows + ip netns exec bob1 ping6 -c 20 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes 64 bytes from fd01::2: icmp_seq=1 ttl=62 time=1.70 ms 64 bytes from fd01::2: icmp_seq=2 ttl=62 time=1.25 ms 64 bytes from fd01::2: icmp_seq=3 ttl=62 time=0.932 ms 64 bytes from fd01::2: icmp_seq=4 ttl=62 time=0.940 ms 64 bytes from fd01::2: icmp_seq=5 ttl=62 time=0.852 ms 64 bytes from fd01::2: icmp_seq=6 ttl=62 time=0.922 ms 64 bytes from fd01::2: icmp_seq=7 ttl=62 time=0.880 ms 64 bytes from fd01::2: icmp_seq=8 ttl=62 time=0.855 ms 64 bytes from fd01::2: icmp_seq=9 ttl=62 time=0.885 ms 64 bytes from fd01::2: icmp_seq=10 ttl=62 time=0.781 ms 64 bytes from fd01::2: icmp_seq=11 ttl=62 time=0.849 ms 64 bytes from fd01::2: icmp_seq=12 ttl=62 time=0.830 ms 64 bytes from fd01::2: icmp_seq=13 ttl=62 time=0.848 ms 64 bytes from fd01::2: icmp_seq=14 ttl=62 time=0.895 ms 64 bytes from fd01::2: icmp_seq=15 ttl=62 time=0.836 ms 64 bytes from fd01::2: icmp_seq=16 ttl=62 time=0.851 ms 64 bytes from fd01::2: icmp_seq=17 ttl=62 time=0.902 ms 64 bytes from fd01::2: icmp_seq=18 ttl=62 time=0.893 ms 64 bytes from fd01::2: icmp_seq=19 ttl=62 time=0.841 ms 64 bytes from fd01::2: icmp_seq=20 ttl=62 time=0.855 ms --- fd01::2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 5719ms rtt min/avg/max/mdev = 0.781/0.930/1.705/0.200 ms <=== ping passed
also verified on ovn2.13-20.12.0-140.el8fdp.x86_64 : + ip netns exec bob1 ping6 -c 2 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes 64 bytes from fd01::2: icmp_seq=1 ttl=62 time=5.20 ms 64 bytes from fd01::2: icmp_seq=2 ttl=62 time=1.50 ms --- fd01::2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 301ms rtt min/avg/max/mdev = 1.495/3.349/5.204/1.855 ms + ovs-appctl dpctl/del-flows + ip netns exec bob1 ping6 -c 20 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes 64 bytes from fd01::2: icmp_seq=1 ttl=62 time=1.85 ms 64 bytes from fd01::2: icmp_seq=2 ttl=62 time=1.40 ms 64 bytes from fd01::2: icmp_seq=3 ttl=62 time=0.998 ms 64 bytes from fd01::2: icmp_seq=4 ttl=62 time=0.971 ms 64 bytes from fd01::2: icmp_seq=5 ttl=62 time=0.963 ms 64 bytes from fd01::2: icmp_seq=6 ttl=62 time=0.987 ms 64 bytes from fd01::2: icmp_seq=7 ttl=62 time=0.975 ms 64 bytes from fd01::2: icmp_seq=8 ttl=62 time=0.991 ms 64 bytes from fd01::2: icmp_seq=9 ttl=62 time=0.984 ms 64 bytes from fd01::2: icmp_seq=10 ttl=62 time=0.883 ms 64 bytes from fd01::2: icmp_seq=11 ttl=62 time=0.935 ms 64 bytes from fd01::2: icmp_seq=12 ttl=62 time=0.977 ms 64 bytes from fd01::2: icmp_seq=13 ttl=62 time=0.917 ms 64 bytes from fd01::2: icmp_seq=14 ttl=62 time=0.919 ms 64 bytes from fd01::2: icmp_seq=15 ttl=62 time=0.989 ms 64 bytes from fd01::2: icmp_seq=16 ttl=62 time=0.926 ms 64 bytes from fd01::2: icmp_seq=17 ttl=62 time=1.03 ms 64 bytes from fd01::2: icmp_seq=18 ttl=62 time=0.982 ms 64 bytes from fd01::2: icmp_seq=19 ttl=62 time=0.959 ms 64 bytes from fd01::2: icmp_seq=20 ttl=62 time=1.02 ms --- fd01::2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 5720ms rtt min/avg/max/mdev = 0.883/1.032/1.848/0.214 ms [root@wsfd-advnetlab19 bz1959008]# rpm -qa | grep -E "openvswitch2.15|ovn2.13" ovn2.13-central-20.12.0-140.el8fdp.x86_64 ovn2.13-20.12.0-140.el8fdp.x86_64 openvswitch2.15-2.15.0-26.el8fdp.x86_64 ovn2.13-host-20.12.0-140.el8fdp.x86_64
also verified on ovn-2021-21.06.0-4.el8fdp.x86_64: + ip netns exec bob1 ping6 -c 2 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes 64 bytes from fd01::2: icmp_seq=1 ttl=62 time=5.28 ms 64 bytes from fd01::2: icmp_seq=2 ttl=62 time=1.65 ms --- fd01::2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 301ms rtt min/avg/max/mdev = 1.654/3.465/5.276/1.811 ms + ovs-appctl dpctl/del-flows + ip netns exec bob1 ping6 -c 20 -i 0.3 -w 15 fd01::2 PING fd01::2(fd01::2) 56 data bytes 64 bytes from fd01::2: icmp_seq=1 ttl=62 time=1.77 ms 64 bytes from fd01::2: icmp_seq=2 ttl=62 time=1.29 ms 64 bytes from fd01::2: icmp_seq=3 ttl=62 time=1.04 ms 64 bytes from fd01::2: icmp_seq=4 ttl=62 time=0.933 ms 64 bytes from fd01::2: icmp_seq=5 ttl=62 time=0.924 ms 64 bytes from fd01::2: icmp_seq=6 ttl=62 time=0.897 ms 64 bytes from fd01::2: icmp_seq=7 ttl=62 time=0.954 ms 64 bytes from fd01::2: icmp_seq=8 ttl=62 time=0.880 ms 64 bytes from fd01::2: icmp_seq=9 ttl=62 time=0.881 ms 64 bytes from fd01::2: icmp_seq=10 ttl=62 time=0.861 ms 64 bytes from fd01::2: icmp_seq=11 ttl=62 time=0.929 ms 64 bytes from fd01::2: icmp_seq=12 ttl=62 time=0.944 ms 64 bytes from fd01::2: icmp_seq=13 ttl=62 time=0.901 ms 64 bytes from fd01::2: icmp_seq=14 ttl=62 time=0.948 ms 64 bytes from fd01::2: icmp_seq=15 ttl=62 time=0.947 ms 64 bytes from fd01::2: icmp_seq=16 ttl=62 time=0.972 ms 64 bytes from fd01::2: icmp_seq=17 ttl=62 time=0.991 ms 64 bytes from fd01::2: icmp_seq=18 ttl=62 time=1.09 ms 64 bytes from fd01::2: icmp_seq=19 ttl=62 time=1.10 ms 64 bytes from fd01::2: icmp_seq=20 ttl=62 time=1.02 ms --- fd01::2 ping statistics --- 20 packets transmitted, 20 received, 0% packet loss, time 5731ms rtt min/avg/max/mdev = 0.861/1.013/1.774/0.201 ms [root@wsfd-advnetlab19 bz1959008]# rpm -qa | grep -E "openvswitch2.15|ovn-2021" ovn-2021-host-21.06.0-4.el8fdp.x86_64 openvswitch2.15-2.15.0-26.el8fdp.x86_64 ovn-2021-central-21.06.0-4.el8fdp.x86_64 ovn-2021-21.06.0-4.el8fdp.x86_64
Hello, The customer is having the following queries, It will be great if we can get answers soon. 1. Can we expect a fix soon? 2. Is the fix be back ported to 4.7? If so, which build? 3. If not, which release and build, we can expect this fix? Regards Sphoorthi
Hey Dhruv, the corresponding OVN-kube bug tracks which release it lands in for OCP: https://bugzilla.redhat.com/show_bug.cgi?id=1958375 (you can see it in the "blocks" field in the bz). Currently that bug points to a fix in 4.9 which is ON_QA. After it is verified we will backport with a new bug in 4.8z, and then to 4.7z.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2971