Description of problem: For a default OVN deployment with 16.1.6, ovn does not respond to traceroute for floating IPs correctly example: openstack server list │ │ +--------------------------------------+-------+--------+----------------------------------+--------+--------+ │ │ | ID | Name | Status | Networks | Image | Flavor | │ │ +--------------------------------------+-------+--------+----------------------------------+--------+--------+ │ │ | 7968a375-ba5a-4643-acfd-697eb8601fdb | test2 | ACTIVE | net1=10.10.10.157, 192.168.2.163 | cirros | | │ │ +--------------------------------------+-------+--------+----------------------------------+--------+--------+ # traceroute -I -n 192.168.2.163 traceroute to 192.168.2.163 (192.168.2.163), 30 hops max, 60 byte packets 1 192.168.0.1 2.797 ms 3.635 ms 4.524 ms 2 * * * 3 192.168.2.163 10.820 ms 10.842 ms 10.828 ms Here 192.168.0.1 is a physical router. The intermediate missing packets "* * *" are missing from ovn it seems. Version-Release number of selected component (if applicable): ovn2.13-20.12.0-135.el8fdp.x86_64 How reproducible: 100% Steps to Reproduce: 1. default OSP 16.1 deployment with ovn 2. create neutron router, instance,& floating ip 3. test with traceroute as outlined above Actual results: Missing "ICMP time exceeded in-transit" from ovn Expected results: OVN send "ICMP time exceeded in-transit" Additional info: I'll add additonal detail in private comments
upstream fix: https://patchwork.ozlabs.org/project/ovn/patch/b4f5456146bf1fa805deb778bd1dca989ba20732.1636583678.git.lorenzo.bianconi@redhat.com/
tested with following script: server: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.181.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.181.25 systemctl restart ovn-controller ovs-vsctl add-br br-phy ovs-vsctl add-port br-phy ens1f1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phy ovn-nbctl ls-add ls1 ovn-nbctl lsp-add ls1 ls1p1 ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1" ovn-nbctl lsp-add ls1 ls1p2 ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2 2001::2" ovn-nbctl lr-add lr1 ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64 ovn-nbctl lsp-add ls1 ls1-lr1 ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01" ovn-nbctl lsp-set-type ls1-lr1 router ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1 ovn-nbctl lrp-add lr1 lr1-pub 00:00:00:00:00:02 172.17.1.254/24 7011::a/64 ovn-nbctl lrp-set-gateway-chassis lr1-pub hv0 ovn-nbctl lr-route-add lr1 0.0.0.0/0 172.17.1.100 lr1-pub ovn-nbctl lr-route-add lr1 ::/0 7011::100 lr1-pub ovn-nbctl ls-add pub ovn-nbctl lsp-add pub pub-lr1 ovn-nbctl lsp-set-type pub-lr1 router ovn-nbctl lsp-set-addresses pub-lr1 router ovn-nbctl lsp-set-options pub-lr1 router-port=lr1-pub ovn-nbctl lsp-add pub ln0 ovn-nbctl lsp-set-type ln0 localnet ovn-nbctl lsp-set-options ln0 network_name=phys ovn-nbctl lsp-set-addresses ln0 unknown ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.17.1.11 192.168.1.1 ls1p1 00:00:00:00:00:02 ovn-nbctl lr-nat-add lr1 dnat_and_snat 7011::11 2001::1 ls1p1 00:00:00:00:00:02 ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1 ip netns add ls1p1 ip link set ls1p1 netns ls1p1 ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01 ip netns exec ls1p1 ip link set ls1p1 up ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1 ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1 ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1 ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1 client: systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.181.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.181.26 systemctl restart ovn-controller ovs-vsctl add-br br-phy ovs-vsctl add-port br-phy ens1f1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phy ovs-vsctl add-port br-phy ext2 -- set interface ext2 type=internal ip netns add ext2 ip link set ext2 netns ext2 ip netns exec ext2 ip link set ext2 up ip netns exec ext2 ip addr add 172.17.1.102/24 dev ext2 ip netns exec ext2 ip -6 addr add 7011::102/64 dev ext2 ip netns exec ext2 ping 172.17.1.11 -c 1 ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 ip netns exec ext2 ping6 7011::11 -c 1 ip netns exec ext2 traceroute6 -I -z 1 -n 7011::11 result on ovn2.13-20.12.0-135: [root@wsfd-advnetlab17 bz2006349]# rpm -qa | grep -E "openvswitch2.15|ovn2.13" ovn2.13-central-20.12.0-135.el8fdp.x86_64 python3-openvswitch2.15-2.15.0-53.el8fdp.x86_64 ovn2.13-20.12.0-135.el8fdp.x86_64 ovn2.13-host-20.12.0-135.el8fdp.x86_64 openvswitch2.15-2.15.0-53.el8fdp.x86_64 + ip netns exec ext2 ping 172.17.1.11 -c 1 PING 172.17.1.11 (172.17.1.11) 56(84) bytes of data. 64 bytes from 172.17.1.11: icmp_seq=1 ttl=63 time=7.36 ms --- 172.17.1.11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.356/7.356/7.356/0.000 ms + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 * * 172.17.1.254 1.261 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * 172.17.1.11 1.582 ms + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 * * * 2 172.17.1.11 0.917 ms 0.238 ms 0.195 ms + ip netns exec ext2 ping6 7011::11 -c 1 PING 7011::11(7011::11) 56 data bytes 64 bytes from 7011::11: icmp_seq=1 ttl=63 time=6.01 ms --- 7011::11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 6.013/6.013/6.013/0.000 ms + ip netns exec ext2 traceroute6 -I -z 1 -n 7011::11 traceroute to 7011::11 (7011::11), 30 hops max, 80 byte packets 1 * * * 2 7011::11 1.565 ms 0.287 ms 0.239 ms result on ovn2.13-20.12.0-191: [root@wsfd-advnetlab17 bz2006349]# rpm -qa | grep -E "openvswitch2.15|ovn2.13" ovn2.13-central-20.12.0-191.el8fdp.x86_64 ovn2.13-20.12.0-191.el8fdp.x86_64 ovn2.13-host-20.12.0-191.el8fdp.x86_64 python3-openvswitch2.15-2.15.0-53.el8fdp.x86_64 openvswitch2.15-2.15.0-53.el8fdp.x86_64 + ip netns exec ext2 ping 172.17.1.11 -c 1 PING 172.17.1.11 (172.17.1.11) 56(84) bytes of data. 64 bytes from 172.17.1.11: icmp_seq=1 ttl=63 time=7.52 ms --- 172.17.1.11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.522/7.522/7.522/0.000 ms + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 172.17.1.11 1.277 ms 1.067 ms 1.323 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * 172.17.1.11 1.548 ms + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 172.17.1.11 1.510 ms 1.051 ms 0.944 ms 2 172.17.1.11 1.025 ms 0.258 ms 1.091 ms + ip netns exec ext2 ping6 7011::11 -c 1 PING 7011::11(7011::11) 56 data bytes 64 bytes from 7011::11: icmp_seq=1 ttl=63 time=5.76 ms --- 7011::11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.756/5.756/5.756/0.000 ms + ip netns exec ext2 traceroute6 -I -z 1 -n 7011::11 traceroute to 7011::11 (7011::11), 30 hops max, 80 byte packets 1 7011::11 1.300 ms 0.917 ms 1.072 ms 2 7011::11 1.551 ms 0.263 ms 0.233 ms questions about the result: the ip for the first hop is the FIP, shouldn't it be the ip for the router? + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 172.17.1.11 1.510 ms 1.051 ms 0.944 ms 2 172.17.1.11 1.025 ms 0.258 ms 1.091 ms the packet on ext2: [root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v dropped privs to tcpdump tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes 22:00:37.462881 f2:67:b5:50:73:e2 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 60624, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 1, length 40 22:00:37.464200 00:00:00:00:00:02 > f2:67:b5:50:73:e2, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 88) 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 (tos 0x0, ttl 1, id 60624, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 1, length 40 22:00:38.464473 f2:67:b5:50:73:e2 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 61350, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 2, length 40 22:00:38.465225 00:00:00:00:00:02 > f2:67:b5:50:73:e2, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 88) 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 (tos 0x0, ttl 1, id 61350, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 2, length 40 22:00:39.466443 f2:67:b5:50:73:e2 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 62177, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 3, length 40 22:00:39.467170 00:00:00:00:00:02 > f2:67:b5:50:73:e2, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 88) 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 (tos 0x0, ttl 1, id 62177, offset 0, flags [none], proto ICMP (1), length 60) <== the source ip of the time exceeded in-transit is the FIP, I don't think it's reasonable 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 3, length 40 22:00:40.468422 f2:67:b5:50:73:e2 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 2, id 63031, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 4, length 40 22:00:40.471291 00:00:00:00:00:02 > f2:67:b5:50:73:e2, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 4629, offset 0, flags [none], proto ICMP (1), length 60) 192.168.1.1 > 172.17.1.102: ICMP echo reply, id 13903, seq 4, length 40 22:00:41.469774 f2:67:b5:50:73:e2 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 2, id 63855, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 5, length 40 22:00:41.471361 00:00:00:00:00:02 > f2:67:b5:50:73:e2, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 4834, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 13903, seq 5, length 40 22:00:42.471661 f2:67:b5:50:73:e2 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 2, id 64705, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 13903, seq 6, length 40 22:00:42.471880 00:00:00:00:00:02 > f2:67:b5:50:73:e2, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 5557, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 13903, seq 6, length 40
use a different mac for FIP, new version script on server : systemctl start openvswitch systemctl start ovn-northd ovn-nbctl set-connection ptcp:6641 ovn-sbctl set-connection ptcp:6642 ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.181.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.181.25 systemctl restart ovn-controller ovs-vsctl add-br br-phy ovs-vsctl add-port br-phy ens1f1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phy ovn-nbctl ls-add ls1 ovn-nbctl lsp-add ls1 ls1p1 ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1" ovn-nbctl lsp-add ls1 ls1p2 ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2 2001::2" ovn-nbctl lr-add lr1 ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64 ovn-nbctl lsp-add ls1 ls1-lr1 ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01" ovn-nbctl lsp-set-type ls1-lr1 router ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1 ovn-nbctl lrp-add lr1 lr1-pub 00:00:00:00:00:02 172.17.1.254/24 7011::a/64 ovn-nbctl lrp-set-gateway-chassis lr1-pub hv0 ovn-nbctl lr-route-add lr1 0.0.0.0/0 172.17.1.100 lr1-pub ovn-nbctl lr-route-add lr1 ::/0 7011::100 lr1-pub ovn-nbctl ls-add pub ovn-nbctl lsp-add pub pub-lr1 ovn-nbctl lsp-set-type pub-lr1 router ovn-nbctl lsp-set-addresses pub-lr1 router ovn-nbctl lsp-set-options pub-lr1 router-port=lr1-pub ovn-nbctl lsp-add pub ln0 ovn-nbctl lsp-set-type ln0 localnet ovn-nbctl lsp-set-options ln0 network_name=phys ovn-nbctl lsp-set-addresses ln0 unknown ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.17.1.11 192.168.1.1 ls1p1 00:00:00:00:00:11 ovn-nbctl lr-nat-add lr1 dnat_and_snat 7011::11 2001::1 ls1p1 00:00:00:00:00:11 ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1 ip netns add ls1p1 ip link set ls1p1 netns ls1p1 ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01 ip netns exec ls1p1 ip link set ls1p1 up ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1 ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1 ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1 ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1 output: [root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp dropped privs to tcpdump tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes 06:59:38.001793 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 253, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 1, length 40 06:59:38.002851 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 88) 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 (tos 0x0, ttl 1, id 253, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 1, length 40 06:59:39.003221 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 898, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 2, length 40 06:59:39.004398 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 88) 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 (tos 0x0, ttl 1, id 898, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 2, length 40 06:59:40.004754 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 1182, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 3, length 40 06:59:40.005729 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto ICMP (1), length 88) 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 (tos 0x0, ttl 1, id 1182, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 3, length 40 06:59:41.006032 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 2, id 1877, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 4, length 40 06:59:41.006267 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 25373, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 40413, seq 4, length 40 06:59:42.007508 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 2, id 2656, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 5, length 40 06:59:42.007718 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 25892, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 40413, seq 5, length 40 06:59:43.009076 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 2, id 3523, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 6, length 40 06:59:43.009265 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 26181, offset 0, flags [none], proto ICMP (1), length 60) 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 40413, seq 6, length 40 [root@wsfd-advnetlab17 bz2006349]# ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 172.17.1.11 1.085 ms 1.219 ms 1.000 ms 2 172.17.1.11 0.258 ms 0.235 ms 0.218 ms
(In reply to Jianlin Shi from comment #8) > use a different mac for FIP, new version script on server : > > systemctl start openvswitch > > systemctl start ovn-northd > ovn-nbctl set-connection ptcp:6641 > > ovn-sbctl set-connection ptcp:6642 > ovs-vsctl set open . external_ids:system-id=hv1 > external_ids:ovn-remote=tcp:20.0.181.25:6642 > external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.181.25 > systemctl restart ovn-controller > > > > ovs-vsctl add-br br-phy > > ovs-vsctl add-port br-phy ens1f1 > > ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phy > > > > ovn-nbctl ls-add ls1 > > ovn-nbctl lsp-add ls1 ls1p1 > > ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1" > > ovn-nbctl lsp-add ls1 ls1p2 > > ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2 2001::2" > > > > ovn-nbctl lr-add lr1 > > ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64 > > ovn-nbctl lsp-add ls1 ls1-lr1 > > ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01" > > ovn-nbctl lsp-set-type ls1-lr1 router > > ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1 > > > > ovn-nbctl lrp-add lr1 lr1-pub 00:00:00:00:00:02 172.17.1.254/24 7011::a/64 > > ovn-nbctl lrp-set-gateway-chassis lr1-pub hv0 > > ovn-nbctl lr-route-add lr1 0.0.0.0/0 172.17.1.100 lr1-pub > > ovn-nbctl lr-route-add lr1 ::/0 7011::100 lr1-pub > > > > ovn-nbctl ls-add pub > > ovn-nbctl lsp-add pub pub-lr1 > ovn-nbctl lsp-set-type pub-lr1 router > > ovn-nbctl lsp-set-addresses pub-lr1 router > > ovn-nbctl lsp-set-options pub-lr1 router-port=lr1-pub > > > > ovn-nbctl lsp-add pub ln0 > > ovn-nbctl lsp-set-type ln0 localnet > > ovn-nbctl lsp-set-options ln0 network_name=phys > > ovn-nbctl lsp-set-addresses ln0 unknown > > > > ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.17.1.11 192.168.1.1 ls1p1 > 00:00:00:00:00:11 > ovn-nbctl lr-nat-add lr1 dnat_and_snat 7011::11 2001::1 ls1p1 > 00:00:00:00:00:11 > > > ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal > external_ids:iface-id=ls1p1 > > > ip netns add ls1p1 > > ip link set ls1p1 netns ls1p1 > > ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01 > > ip netns exec ls1p1 ip link set ls1p1 up > > ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1 > > ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1 > > ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1 > > ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1 > > > output: > > [root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp > > dropped privs to tcpdump > tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 > bytes > 06:59:38.001793 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 1, id 253, offset 0, flags [none], proto > ICMP (1), length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 1, length > 40 > 06:59:38.002851 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 > (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto > ICMP (1), length 88) > 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 > (tos 0x0, ttl 1, id 253, offset 0, flags [none], proto ICMP (1), > length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 1, length > 40 > 06:59:39.003221 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 1, id 898, offset 0, flags [none], proto > ICMP (1), length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 2, length 40 > 06:59:39.004398 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 > (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto > ICMP (1), length 88) > 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 > > (tos 0x0, ttl 1, id 898, offset 0, flags [none], proto ICMP (1), > length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 2, length > 40 > 06:59:40.004754 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 1, id 1182, offset 0, flags [none], proto > ICMP (1), length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 3, length > 40 > 06:59:40.005729 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 > (0x0800), length 102: (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto > ICMP (1), length 88) > 172.17.1.11 > 172.17.1.102: ICMP time exceeded in-transit, length 68 > (tos 0x0, ttl 1, id 1182, offset 0, flags [none], proto ICMP (1), > length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 3, length > 40 > 06:59:41.006032 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 2, id 1877, offset 0, flags [none], proto > ICMP (1), length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 4, length > 40 > 06:59:41.006267 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 25373, offset 0, flags [none], > proto ICMP (1), length 60) > 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 40413, seq 4, length 40 > 06:59:42.007508 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 2, id 2656, offset 0, flags [none], proto > ICMP (1), length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 5, length > 40 > 06:59:42.007718 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 25892, offset 0, flags [none], > proto ICMP (1), length 60) > 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 40413, seq 5, length 40 > > 06:59:43.009076 52:1f:a2:7e:86:3b > 00:00:00:00:00:11, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 2, id 3523, offset 0, flags [none], proto > ICMP (1), length 60) > 172.17.1.102 > 172.17.1.11: ICMP echo request, id 40413, seq 6, length 40 > 06:59:43.009265 00:00:00:00:00:11 > 52:1f:a2:7e:86:3b, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 26181, offset 0, flags [none], > proto ICMP (1), length 60) > 172.17.1.11 > 172.17.1.102: ICMP echo reply, id 40413, seq 6, length 40 > > [root@wsfd-advnetlab17 bz2006349]# ip netns exec ext2 traceroute -I -z 1 -n > 172.17.1.11 > traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets > 1 172.17.1.11 1.085 ms 1.219 ms 1.000 ms > 2 172.17.1.11 0.258 ms 0.235 ms 0.218 ms Since you are pinging the FIP (172.17.1.11) you will resolve the FIP mac address (00:00:00:00:00:11) and you will communicate directly with FIP (w/o sending traffic through the logical router lr1). The first line refers to ICMP req with TTL 1 that is discarded by ovn. In this case the FIP will send the ICMP error message so traceroute will show 172.17.1.11 as source IP. The second line refers to the ICMP traffic with TTL 2 that is properly received by the destination (with IP 172.17.1.11). Do you think it is weird the src IP of the ICMP error messages is 172.17.1.11?
(In reply to lorenzo bianconi from comment #9) > > [root@wsfd-advnetlab17 bz2006349]# ip netns exec ext2 traceroute -I -z 1 -n > > 172.17.1.11 > > traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets > > 1 172.17.1.11 1.085 ms 1.219 ms 1.000 ms > > 2 172.17.1.11 0.258 ms 0.235 ms 0.218 ms > > Since you are pinging the FIP (172.17.1.11) you will resolve the FIP mac > address (00:00:00:00:00:11) and you will communicate directly with FIP (w/o > sending traffic through the logical router lr1). > The first line refers to ICMP req with TTL 1 that is discarded by ovn. In > this case the FIP will send the ICMP error message so traceroute will show > 172.17.1.11 as source IP. > The second line refers to the ICMP traffic with TTL 2 that is properly > received by the destination (with IP 172.17.1.11). > Do you think it is weird the src IP of the ICMP error messages is > 172.17.1.11? yes, and the 2 hops in the output of traceroute are 172.17.1.11, that is weird.
(In reply to Jianlin Shi from comment #10) > (In reply to lorenzo bianconi from comment #9) > > > > [root@wsfd-advnetlab17 bz2006349]# ip netns exec ext2 traceroute -I -z 1 -n > > > 172.17.1.11 > > > traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets > > > 1 172.17.1.11 1.085 ms 1.219 ms 1.000 ms > > > 2 172.17.1.11 0.258 ms 0.235 ms 0.218 ms > > > > Since you are pinging the FIP (172.17.1.11) you will resolve the FIP mac > > address (00:00:00:00:00:11) and you will communicate directly with FIP (w/o > > sending traffic through the logical router lr1). > > The first line refers to ICMP req with TTL 1 that is discarded by ovn. In > > this case the FIP will send the ICMP error message so traceroute will show > > 172.17.1.11 as source IP. > > The second line refers to the ICMP traffic with TTL 2 that is properly > > received by the destination (with IP 172.17.1.11). > > Do you think it is weird the src IP of the ICMP error messages is > > 172.17.1.11? > > yes, and the 2 hops in the output of traceroute are 172.17.1.11, that is > weird. - we have 2 lines in the traceroute output since packets are logically forwarded by the ovn logical router (lr1) and the first three packets are sent by traceroute with TTL = 1 (so discarded by ovn) - since the traffic is managed locally for FIP, we could send the ICMP error messages with src ip set to the logical router ip (172.17.1.254) but ovn will use the FIP mac (00:00:00:00:00:11) as src mac. This can poison the receiver arp cache. We can't send the packet to the hv running the gw-router-port since we will trigger an unbalance in connection-tracking state (the request sent by traceroute is received by the hv running the FIP).
also tested on ovn-2021-21.09.1-23: + ip netns exec ext2 ping 172.17.1.11 -c 1 PING 172.17.1.11 (172.17.1.11) 56(84) bytes of data. 64 bytes from 172.17.1.11: icmp_seq=1 ttl=63 time=7.69 ms --- 172.17.1.11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 7.688/7.688/7.688/0.000 ms + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 172.17.1.11 1.323 ms 1.119 ms 1.302 ms 2 172.17.1.11 1.318 ms 0.651 ms 0.247 ms + ip netns exec ext2 traceroute -I -z 1 -n 172.17.1.11 traceroute to 172.17.1.11 (172.17.1.11), 30 hops max, 60 byte packets 1 172.17.1.11 1.076 ms 1.113 ms 1.157 ms 2 172.17.1.11 0.210 ms 0.209 ms 0.195 ms + ip netns exec ext2 ping6 7011::11 -c 1 PING 7011::11(7011::11) 56 data bytes 64 bytes from 7011::11: icmp_seq=1 ttl=63 time=5.68 ms --- 7011::11 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 5.681/5.681/5.681/0.000 ms + ip netns exec ext2 traceroute6 -I -z 1 -n 7011::11 traceroute to 7011::11 (7011::11), 30 hops max, 80 byte packets 1 7011::11 1.240 ms 1.162 ms 1.094 ms 2 7011::11 1.762 ms 0.706 ms 0.264 ms [root@wsfd-advnetlab17 bz2006349]# rpm -qa | grep -E "openvswitch2.15|ovn-2021" ovn-2021-21.09.1-23.el8fdp.x86_64 python3-openvswitch2.15-2.15.0-53.el8fdp.x86_64 ovn-2021-central-21.09.1-23.el8fdp.x86_64 openvswitch2.15-2.15.0-53.el8fdp.x86_64 ovn-2021-host-21.09.1-23.el8fdp.x86_64 set VERIFIED per comment 11
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0049