Bug 2021713
| Summary: | tcp reset doesn't go back on the same way as the original packet when --ecmp-symmetric-reply for the route | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Jianlin Shi <jishi> |
| Component: | ovn-2021 | Assignee: | lorenzo bianconi <lorenzo.bianconi> |
| Status: | CLOSED NOTABUG | QA Contact: | Jianlin Shi <jishi> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | FDP 20.I | CC: | ctrautma, jiji, lorenzo.bianconi |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-07-13 08:07:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
the bug can't be reproduced on rhel-8.2.0 with the ovn version described in the Description and with the reproducer in the Description. bug the issue didn't occur when I upgraded the kernel to 4.18.0-501.el8 even after run the reproducer for 3 hours. and lorenzo bianconi also tried and got the same result close the bug as NOTABUG |
Description of problem: tcp reset doesn't go back on the same way as the original packet when --ecmp-symmetric-reply for the route Version-Release number of selected component (if applicable): ovn-2021-21.09.0-20 How reproducible: Always Steps to Reproduce: 1. setup env # foo -- R1 -- join - R2 -- alice -- | # | | server # bar ---- - R3 --- bob ---- | # systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.178.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.178.25 systemctl restart ovn-controller ovn-nbctl lr-add R1 ovn-nbctl lr-add R2 ovn-nbctl lr-add R3 ovn-nbctl set logical_router R1 options:chassis=hv1 ovn-nbctl set logical_router R2 options:chassis=hv1 ovn-nbctl set logical_router R3 options:chassis=hv1 ovn-nbctl ls-add foo ovn-nbctl ls-add bar ovn-nbctl ls-add alice ovn-nbctl ls-add bob ovn-nbctl ls-add join ovn-nbctl lrp-add R1 foo 00:00:01:01:02:03 192.168.1.1/24 2001::1/64 ovn-nbctl lsp-add foo rp-foo -- set logical_switch_port rp-foo \ type=router options:router-port=foo addresses=\"00:00:01:01:02:03\" ovn-nbctl lrp-add R1 bar 00:00:01:01:02:04 192.168.2.1/24 2002::1/64 ovn-nbctl lsp-add bar rp-bar -- set Logical_Switch_Port rp-bar \ type=router options:router-port=bar addresses=\"00:00:01:01:02:04\" ovn-nbctl lrp-add R2 alice 00:00:02:01:02:03 172.16.1.1/24 3001::1/64 ovn-nbctl lsp-add alice rp-alice -- set Logical_Switch_Port rp-alice \ type=router options:router-port=alice addresses=\"00:00:02:01:02:03\" ovn-nbctl lrp-add R3 bob 00:00:03:01:02:03 172.17.1.1/24 3002::1/64 ovn-nbctl lsp-add bob rp-bob -- set Logical_Switch_Port rp-bob \ type=router options:router-port=bob addresses=\"00:00:03:01:02:03\" ovn-nbctl lrp-add R1 R1_join 00:00:04:01:02:03 20.0.0.1/24 4000::1/64 ovn-nbctl lsp-add join r1-join -- set Logical_Switch_Port r1-join \ type=router options:router-port=R1_join addresses='"00:00:04:01:02:03"' ovn-nbctl lrp-add R2 R2_join 00:00:04:01:02:04 20.0.0.2/24 4000::2/64 ovn-nbctl lsp-add join r2-join -- set Logical_Switch_Port r2-join \ type=router options:router-port=R2_join addresses='"00:00:04:01:02:04"' ovn-nbctl lrp-add R3 R3_join 00:00:04:01:02:05 20.0.0.3/24 4000::3/64 ovn-nbctl lsp-add join r3-join -- set Logical_Switch_Port r3-join \ type=router options:router-port=R3_join addresses='"00:00:04:01:02:05"' ovn-nbctl lr-route-add R2 192.168.0.0/16 20.0.0.1 ovn-nbctl lr-route-add R3 192.168.0.0/16 20.0.0.1 ovn-nbctl lr-route-add R2 2001::/64 4000::1 ovn-nbctl lr-route-add R2 2002::/64 4000::1 ovn-nbctl lr-route-add R3 2001::/64 4000::1 ovn-nbctl lr-route-add R3 2002::/64 4000::1 ovn-nbctl lr-route-add R2 1.1.1.0/24 172.16.1.3 ovn-nbctl lr-route-add R3 1.1.1.0/24 172.17.1.4 ovn-nbctl lr-route-add R2 1111::/64 3001::3 ovn-nbctl lr-route-add R3 1111::/64 3002::4 ip netns add foo1 ovs-vsctl add-port br-int foo1 -- set interface foo1 type=internal ip link set foo1 netns foo1 ip netns exec foo1 ip link set foo1 address f0:00:00:01:02:03 ip netns exec foo1 ip link set foo1 up ip netns exec foo1 ip addr add 192.168.1.2/24 dev foo1 ip netns exec foo1 ip -6 addr add 2001::2/64 dev foo1 ip netns exec foo1 ip route add default via 192.168.1.1 dev foo1 ip netns exec foo1 ip -6 route add default via 2001::1 dev foo1 ovs-vsctl set interface foo1 external_ids:iface-id=foo1 ovn-nbctl lsp-add foo foo1 -- lsp-set-addresses foo1 "f0:00:00:01:02:03 192.168.1.2 2001::2" ip netns add bar1 ip link add bar1 netns bar1 type veth peer name bar1_br ip netns exec bar1 ip link set bar1 address f0:00:00:01:02:05 ip netns exec bar1 ip link set bar1 up ip netns exec bar1 ip addr add 192.168.2.2/24 dev bar1 ip netns exec bar1 ip -6 addr add 2002::2/64 dev bar1 ip netns exec bar1 ip route add default via 192.168.2.1 dev bar1 ip netns exec bar1 ip -6 route add default via 2002::1 dev bar1 ip link set bar1_br up ovs-vsctl add-port br-int bar1_br ovs-vsctl set interface bar1_br external_ids:iface-id=bar1 ovn-nbctl lsp-add bar bar1 -- lsp-set-addresses bar1 "f0:00:00:01:02:05 192.168.2.2 2002::2" ovs-vsctl add-br br_alice ovs-vsctl add-br br_bob ovs-vsctl set open . external-ids:ovn-bridge-mappings=net_alice:br_alice,net_bob:br_bob ovn-nbctl lsp-add alice ln_alice ovn-nbctl lsp-set-type ln_alice localnet ovn-nbctl lsp-set-addresses ln_alice unknown ovn-nbctl lsp-set-options ln_alice network_name=net_alice ip netns add alice1 ovs-vsctl add-port br_alice alice1 -- set interface alice1 type=internal ip link set alice1 netns alice1 ip netns exec alice1 ip link set alice1 address f0:00:00:01:02:04 ip netns exec alice1 ip link set alice1 up ip netns exec alice1 ip addr add 172.16.1.3/24 dev alice1 ip netns exec alice1 ip -6 addr add 3001::3/64 dev alice1 ip netns exec alice1 ip route add default via 172.16.1.1 dev alice1 ip netns exec alice1 ip -6 route add default via 3001::1 dev alice1 ovn-nbctl lsp-add bob ln_bob ovn-nbctl lsp-set-type ln_bob localnet ovn-nbctl lsp-set-addresses ln_bob unknown ovn-nbctl lsp-set-options ln_bob network_name=net_bob ip netns add bob1 ip link add bob1 netns bob1 type veth peer name bob1_br ip netns exec bob1 ip link set bob1 address f0:00:00:01:02:06 ip netns exec bob1 ip link set bob1 up ip netns exec bob1 ip addr add 172.17.1.4/24 dev bob1 ip netns exec bob1 ip -6 addr add 3002::4/64 dev bob1 ip netns exec bob1 ip route add default via 172.17.1.1 dev bob1 ip netns exec bob1 ip -6 route add default via 3002::1 dev bob1 ip link set bob1_br up ovs-vsctl add-port br_bob bob1_br ip link add br_test type bridge ip link set br_test up ip link add a1 netns alice1 type veth peer name a1_br ip link add b1 netns bob1 type veth peer name b1_br ip link set a1_br master br_test ip link set b1_br master br_test ip link set a1_br up ip link set b1_br up ip netns exec alice1 ip link set a1 up ip netns exec bob1 ip link set b1 up ip netns exec alice1 ip addr add 1.1.1.1/24 dev a1 ip netns exec alice1 ip -6 addr add 1111::1/64 dev a1 ip netns exec bob1 ip addr add 1.1.1.2/24 dev b1 ip netns exec bob1 ip -6 addr add 1111::2/64 dev b1 ip netns exec alice1 sysctl -w net.ipv4.conf.all.forwarding=1 ip netns exec bob1 sysctl -w net.ipv4.conf.all.forwarding=1 ip netns exec alice1 sysctl -w net.ipv6.conf.all.forwarding=1 ip netns exec bob1 sysctl -w net.ipv6.conf.all.forwarding=1 ip netns add server ip link add s1 netns server type veth peer name s1_br ip link set s1_br master br_test ip link set s1_br up ip netns exec server ip link set s1 up ip netns exec server ip addr add 1.1.1.10/24 dev s1 ip netns exec server ip route add default via 1.1.1.1 dev s1 ip netns exec server ip -6 addr add 1111::10/64 dev s1 ip netns exec server ip -6 route add default via 1111::1 dev s1 ip netns exec server sysctl -w net.ipv4.conf.all.rp_filter=0 ip netns exec server sysctl -w net.ipv4.conf.default.rp_filter=0 ovn-nbctl --ecmp-symmetric-reply lr-route-add R1 0.0.0.0/0 20.0.0.2 ovn-nbctl --ecmp-symmetric-reply lr-route-add R1 0.0.0.0/0 20.0.0.3 ovn-nbctl --ecmp-symmetric-reply lr-route-add R1 ::/0 4000::2 ovn-nbctl --ecmp-symmetric-reply lr-route-add R1 ::/0 4000::3 2. run tcp ip netns exec foo1 nc -4 -l 10010 -k & ip netns exec foo1 nc -6 -l 10011 -k & ip netns exec bar1 nc -4 -l 10010 -k & ip netns exec bar1 nc -6 -l 10011 -k & while : do ip netns exec foo1 tcpdump -i foo1 -w foo1.pcap -nnle & ip netns exec bar1 tcpdump -i bar1 -w bar1.pcap -nnle & ip netns exec server tcpdump -i s1 -w s1.pcap -nnle & ip netns exec bob1 tcpdump -U -i bob1 host 192.168.1.2 -nnle -v &> bob1_ip4.log & tcpdump_pid=$! sleep 2 for i in {1..10} do ip netns exec server nc 192.168.1.2 10010 <<< h done kill $tcpdump_pid sleep 2 if grep 192.168.1.2 bob1_ip4.log then break fi ip netns exec bob1 tcpdump -U -i bob1 host 2001::2 -nnle -v &> bob1_ip6.log & tcpdump_pid=$! sleep 5 for i in {1..10} do ip netns exec server nc 2001::2 10011 <<< h done kill $tcpdump_pid sleep 5 if grep 2001::2 bob1_ip6.log then break fi ip netns exec bob1 tcpdump -U -i bob1 host 192.168.2.2 -nnle -v &> bob1_ip42.log & tcpdump_pid=$! sleep 5 for i in {1..10} do ip netns exec server nc 192.168.2.2 10010 <<< h done kill $tcpdump_pid sleep 5 if grep 192.168.2.2 bob1_ip42.log then break fi ip netns exec bob1 tcpdump -U -i bob1 host 2002::2 -nnle -v &> bob1_ip62.log & tcpdump_pid=$! sleep 5 for i in {1..10} do ip netns exec server nc 2002::2 10011 <<< h done kill $tcpdump_pid sleep 5 if grep 2002::2 bob1_ip62.log then break fi pkill tcpdump sleep 2 tcpdump -r bar1.pcap -nnle | grep -w R tcpdump -r foo1.pcap -nnle | grep -w R done ip netns exec foo1 pkill nc ip netns exec bar1 pkill nc Actual results: + grep 2001::2 bob1_ip6.log 21:40:52.655848 00:00:03:01:02:03 > f0:00:00:01:02:06, ethertype IPv6 (0x86dd), length 74: (flowlabel 0x7f9ce, hlim 62, next-header TCP (6) payload length: 20) 2001::2.10011 > 1111::10.40474: Flags [R], cksum 0xe38e (correct), seq 3689544206, win 0, length 0 <=== the tcp reset goes back through R3 -> bob [root@wsfd-advnetlab16 bz1849683]# tcpdump -r foo1.pcap -nnle | grep -w R reading from file foo1.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump 21:40:45.160125 f0:00:00:01:02:03 > 00:00:01:01:02:03, ethertype IPv4 (0x0800), length 54: 192.168.1.2.10010 > 1.1.1.10.43824: Flags [R], seq 2702744360, win 0, length 0 21:40:52.645839 f0:00:00:01:02:03 > 00:00:01:01:02:03, ethertype IPv6 (0x86dd), length 74: 2001::2.10011 > 1111::10.40474: Flags [R], seq 3689544206, win 0, length 0 [root@wsfd-advnetlab16 bz1849683]# tcpdump -r foo1.pcap -nnle port 40474 reading from file foo1.pcap, link-type EN10MB (Ethernet) dropped privs to tcpdump 21:40:52.639948 00:00:01:01:02:03 > f0:00:00:01:02:03, ethertype IPv6 (0x86dd), length 94: 1111::10.40474 > 2001::2.10011: Flags [S], seq 782549477, win 28800, options [mss 1440,sackOK,TS val 1944736763 ecr 0,nop,wscale 7], length 0 21:40:52.640001 f0:00:00:01:02:03 > 00:00:01:01:02:03, ethertype IPv6 (0x86dd), length 94: 2001::2.10011 > 1111::10.40474: Flags [S.], seq 3689544205, ack 782549478, win 28560, options [mss 1440,sackOK,TS val 3842428731 ecr 1944736763,nop,wscale 7], length 0 21:40:52.644556 00:00:01:01:02:03 > f0:00:00:01:02:03, ethertype IPv6 (0x86dd), length 88: 1111::10.40474 > 2001::2.10011: Flags [P.], seq 1:3, ack 1, win 225, options [nop,nop,TS val 1944736771 ecr 3842428731], length 2 21:40:52.644617 f0:00:00:01:02:03 > 00:00:01:01:02:03, ethertype IPv6 (0x86dd), length 86: 2001::2.10011 > 1111::10.40474: Flags [.], ack 3, win 224, options [nop,nop,TS val 3842428735 ecr 1944736771], length 0 21:40:52.644740 00:00:01:01:02:03 > f0:00:00:01:02:03, ethertype IPv6 (0x86dd), length 86: 1111::10.40474 > 2001::2.10011: Flags [F.], seq 3, ack 1, win 225, options [nop,nop,TS val 1944736771 ecr 3842428731], length 0 21:40:52.644840 f0:00:00:01:02:03 > 00:00:01:01:02:03, ethertype IPv6 (0x86dd), length 86: 2001::2.10011 > 1111::10.40474: Flags [F.], seq 1, ack 4, win 224, options [nop,nop,TS val 3842428736 ecr 1944736771], length 0 21:40:52.644951 00:00:01:01:02:03 > f0:00:00:01:02:03, ethertype IPv6 (0x86dd), length 86: 1111::10.40474 > 2001::2.10011: Flags [.], ack 2, win 225, options [nop,nop,TS val 1944736772 ecr 3842428736], length 0 21:40:52.645806 00:00:01:01:02:03 > f0:00:00:01:02:03, ethertype IPv6 (0x86dd), length 86: 1111::10.40474 > 2001::2.10011: Flags [.], ack 1, win 225, options [nop,nop,TS val 1944736771 ecr 3842428731], length 0 21:40:52.645839 f0:00:00:01:02:03 > 00:00:01:01:02:03, ethertype IPv6 (0x86dd), length 74: 2001::2.10011 > 1111::10.40474: Flags [R], seq 3689544206, win 0, length 0 Expected results: tcp reset should not go through R3 -> bob Additional info: [root@wsfd-advnetlab16 bz1849683]# rpm -qa | grep -E "openvswitch2.16|ovn-2021" ovn-2021-central-21.09.0-20.el8fdp.x86_64 openvswitch2.16-2.16.0-25.el8fdp.x86_64 ovn-2021-21.09.0-20.el8fdp.x86_64 ovn-2021-host-21.09.0-20.el8fdp.x86_64 python3-openvswitch2.16-2.16.0-25.el8fdp.x86_64 the issue also exist on ovn-2021-21.06.0-29