Bug 1960096
| Summary: | Wrong source ip is used if an external gateway and an instance with floating ip is located in the same node | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Takashi Kajinami <tkajinam> |
| Component: | ovn2.13 | Assignee: | lorenzo bianconi <lorenzo.bianconi> |
| Status: | CLOSED ERRATA | QA Contact: | Jianlin Shi <jishi> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | FDP 20.I | CC: | apevec, ctrautma, dcbw, ffernand, fleitner, jiji, lhh, lorenzo.bianconi, majopela, mmichels, ralongi, scohen, twilson |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ovn-2021-21.09.1-22.el8fdp-ovn2.13-20.12.0-191.el8fdp | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-01-10 16:49:01 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Additional note:
This issue was not solved even if I disable SNAT of the router.
~~~
05:16:48.705097 fa:16:3e:05:06:44 > 52:54:00:14:90:f3, ethertype IPv4 (0x0800), length 98: 10.0.0.215 > 8.8.8.8: ICMP echo request, id 24324, seq 70, length 64
05:16:48.713064 52:54:00:14:90:f3 > fa:16:3e:97:4f:4e, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.0.215: ICMP echo reply, id 24324, seq 70, length 64
05:16:49.705522 fa:16:3e:05:06:44 > 52:54:00:14:90:f3, ethertype IPv4 (0x0800), length 98: 10.0.0.215 > 8.8.8.8: ICMP echo request, id 24324, seq 71, length 64
05:16:49.713458 52:54:00:14:90:f3 > fa:16:3e:97:4f:4e, ethertype IPv4 (0x0800), length 98: 8.8.8.8 > 10.0.0.215: ICMP echo reply, id 24324, seq 71, length 64
~~~
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack port list --long | egrep -e floating -e gateway
| 96a3eab1-e860-4171-9f3b-18fba0f026f4 | | fa:16:3e:05:06:44 | ip_address='10.0.0.210', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | N/A | None | network:floatingip | |
| d8866825-dba8-407a-bfc5-30a3f24d86b3 | | fa:16:3e:5a:24:55 | ip_address='10.0.0.242', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | N/A | None | network:floatingip | |
| dc427ee9-2e36-43d6-9778-8419a3b14752 | | fa:16:3e:97:4f:4e | ip_address='10.0.0.215', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | ACTIVE | None | network:router_gateway | |
(overcloud) [stack@undercloud-0 ~]$ openstack port show dc427ee9-2e36-43d6-9778-8419a3b14752
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | compute-0.redhat.local |
| binding_profile | |
| binding_vif_details | port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2021-05-13T05:10:31Z |
| data_plane_status | None |
| description | |
| device_id | 6f7b40aa-c1ec-4e07-972d-a71af103db7d |
| device_owner | network:router_gateway |
| dns_assignment | fqdn='host-10-0-0-215.openstacklocal.', hostname='host-10-0-0-215', ip_address='10.0.0.215' |
| | fqdn='host-2620-52-0-13b8--1000-60.openstacklocal.', hostname='host-2620-52-0-13b8--1000-60', ip_address='2620:52:0:13b8::1000:60' |
| dns_domain | None |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.0.215', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' |
| | ip_address='2620:52:0:13b8::1000:60', subnet_id='20ed52a9-1788-4ad8-8e4a-8d0d40e6eb73' |
| id | dc427ee9-2e36-43d6-9778-8419a3b14752 |
| location | cloud='', project.domain_id=, project.domain_name='Default', project.id='4c9a7610e1b043be9ba5fcb530a964ad', project.name='admin', region_name='regionOne', zone= |
| mac_address | fa:16:3e:97:4f:4e |
| name | |
| network_id | 08012876-fb02-4f3e-9000-40810c433c3c |
| port_security_enabled | False |
| project_id | |
| propagate_uplink_status | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 4 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2021-05-13T05:10:39Z |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack router show router
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2021-04-15T06:54:51Z |
| description | |
| external_gateway_info | {"network_id": "08012876-fb02-4f3e-9000-40810c433c3c", "external_fixed_ips": [{"subnet_id": "6661e943-1789-439e-b957-65d93748fa8c", "ip_address": "10.0.0.215"}, {"subnet_id": "20ed52a9-1788-4ad8-8e4a-8d0d40e6eb73", "ip_address": "2620:52:0:13b8::1000:60"}], "enable_snat": false} |
| flavor_id | None |
| id | 6f7b40aa-c1ec-4e07-972d-a71af103db7d |
| interfaces_info | [{"port_id": "3cf4d04e-dfca-4a1a-b72e-56d10d422bc7", "ip_address": "192.168.10.1", "subnet_id": "1e717b5b-68e9-416c-990e-6d34390474bb"}] |
| location | cloud='', project.domain_id=, project.domain_name='Default', project.id='4c9a7610e1b043be9ba5fcb530a964ad', project.name='admin', region_name='regionOne', zone= |
| name | router |
| project_id | 4c9a7610e1b043be9ba5fcb530a964ad |
| revision_number | 25 |
| routes | |
| status | ACTIVE |
| tags | |
| updated_at | 2021-05-13T05:14:39Z |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
~~~
(In reply to Takashi Kajinami from comment #1) > Additional note: > This issue was not solved even if I disable SNAT of the router. It seems there is a bug in networking-ovn and SNAT was not disabled properly. https://bugzilla.redhat.com/show_bug.cgi?id=1962051 So please ignore this. (In reply to Takashi Kajinami from comment #6) > (In reply to Takashi Kajinami from comment #1) > > Additional note: > > This issue was not solved even if I disable SNAT of the router. > > It seems there is a bug in networking-ovn and SNAT was not disabled properly. > https://bugzilla.redhat.com/show_bug.cgi?id=1962051 > > So please ignore this. Sorry if I don't understand. Do you mean that this bug should be closed, or only that your comment "This issue was not solved even if I disable SNAT of the router." should be ignored? Thanks! (In reply to Dan Williams from comment #7) > (In reply to Takashi Kajinami from comment #6) > > (In reply to Takashi Kajinami from comment #1) > > > Additional note: > > > This issue was not solved even if I disable SNAT of the router. > > > > It seems there is a bug in networking-ovn and SNAT was not disabled properly. > > https://bugzilla.redhat.com/show_bug.cgi?id=1962051 > > > > So please ignore this. > > Sorry if I don't understand. Do you mean that this bug should be closed, or > only that your comment "This issue was not solved even if I disable SNAT of > the router." should be ignored? Thanks! In short, no. The issue is still present unless we implement the following workarounds, 1. disable distributed floating ip 2. disable SNAT 2 didn't work in my testing but it turned out that SNAT was not properly disabled because of the bug in networking-ovn. However both of these two workaround causes limitation in efficiency or functionality, and wouldn't be an ideal solution here... upstream fix: commit f100a1216854290ee38867cee9eedb442313a7a6 (HEAD -> main, mainline/main)
Author: Lorenzo Bianconi <lorenzo.bianconi>
Date: Mon Nov 15 17:36:22 2021 +0100
northd: fix FIP traffic with distributed gw router port on the same hv
If the hv has FIP assigned, traffic has to be sent out using the FIP
even if a distributed gw router port is scheduled on the local hv.
In this particular use-case without the proposed patch, the traffic
is sent out with FIP mac but using distributed gw router port IP.
Related bz: https://bugzilla.redhat.com/show_bug.cgi?id=1960096
Signed-off-by: Lorenzo Bianconi <lorenzo.bianconi>
Signed-off-by: Numan Siddique <numans>
Tested with following script:
setup on hv1:
systemctl start openvswitch
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv1 external_ids:ovn-remote=tcp:20.0.181.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.181.25
systemctl restart ovn-controller
ovs-vsctl add-br br-phy
ovs-vsctl add-port br-phy ens1f1
ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phy
ovn-nbctl ls-add ls1
ovn-nbctl lsp-add ls1 ls1p1
ovn-nbctl lsp-set-addresses ls1p1 "00:00:00:01:01:01 192.168.1.1 2001::1"
ovn-nbctl lsp-add ls1 ls1p2
ovn-nbctl lsp-set-addresses ls1p2 "00:00:00:01:01:02 192.168.1.2 2001::2"
ovn-nbctl lr-add lr1
ovn-nbctl lrp-add lr1 lr1-ls1 00:00:00:00:00:01 192.168.1.254/24 2001::a/64
ovn-nbctl lsp-add ls1 ls1-lr1
ovn-nbctl lsp-set-addresses ls1-lr1 "00:00:00:00:00:01"
ovn-nbctl lsp-set-type ls1-lr1 router
ovn-nbctl lsp-set-options ls1-lr1 router-port=lr1-ls1
ovn-nbctl lrp-add lr1 lr1-pub 00:00:00:00:00:02 172.17.1.254/24 7011::a/64
ovn-nbctl lrp-set-gateway-chassis lr1-pub hv1
ovn-nbctl lr-route-add lr1 0.0.0.0/0 172.17.1.100 lr1-pub
ovn-nbctl lr-route-add lr1 ::/0 7011::100 lr1-pub
ovn-nbctl ls-add pub
ovn-nbctl lsp-add pub pub-lr1
ovn-nbctl lsp-set-type pub-lr1 router
ovn-nbctl lsp-set-addresses pub-lr1 router
ovn-nbctl lsp-set-options pub-lr1 router-port=lr1-pub
ovn-nbctl lsp-add pub ln0
ovn-nbctl lsp-set-type ln0 localnet
ovn-nbctl lsp-set-options ln0 network_name=phys
ovn-nbctl lsp-set-addresses ln0 unknown
ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.17.1.11 192.168.1.1 ls1p1 00:00:00:00:ff:02
ovn-nbctl lr-nat-add lr1 dnat_and_snat 7011::11 2001::1 ls1p1 00:00:00:00:ff:02
ovn-nbctl lr-nat-add lr1 snat 172.17.1.41 192.168.1.0/24
ovn-nbctl lr-nat-add lr1 snat 7011::41 2001::/64
ovn-nbctl lr-nat-add lr1 dnat_and_snat 172.17.1.12 192.168.1.2 ls1p2 00:00:00:00:ff:22
ovn-nbctl lr-nat-add lr1 dnat_and_snat 7011::12 2001::2 ls1p2 00:00:00:00:ff:22
ovs-vsctl add-port br-int ls1p1 -- set interface ls1p1 type=internal external_ids:iface-id=ls1p1
ip netns add ls1p1
ip link set ls1p1 netns ls1p1
ip netns exec ls1p1 ip link set ls1p1 address 00:00:00:01:01:01
ip netns exec ls1p1 ip link set ls1p1 up
ip netns exec ls1p1 ip addr add 192.168.1.1/24 dev ls1p1
ip netns exec ls1p1 ip addr add 2001::1/64 dev ls1p1
ip netns exec ls1p1 ip route add default via 192.168.1.254 dev ls1p1
ip netns exec ls1p1 ip -6 route add default via 2001::a dev ls1p1
setup on hv0:
systemctl start openvswitch
systemctl start ovn-northd
ovn-nbctl set-connection ptcp:6641
ovn-sbctl set-connection ptcp:6642
ovs-vsctl set open . external_ids:system-id=hv0 external_ids:ovn-remote=tcp:20.0.181.25:6642 external_ids:ovn-encap-type=geneve external_ids:ovn-encap-ip=20.0.181.26
systemctl restart ovn-controller
ovs-vsctl add-br br-phy
ovs-vsctl add-port br-phy ens1f1
ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phy
ovs-vsctl add-port br-phy ext2 -- set interface ext2 type=internal
ip netns add ext2
ip link set ext2 netns ext2
ip netns exec ext2 ip link set ext2 up
ip netns exec ext2 ip addr add 172.17.1.102/24 dev ext2
ip netns exec ext2 ip -6 addr add 7011::102/64 dev ext2
ovs-vsctl add-port br-int ls1p2 -- set interface ls1p2 type=internal external_ids:iface-id=ls1p2
ip netns add ls1p2
ip link set ls1p2 netns ls1p2
ip netns exec ls1p2 ip link set ls1p2 address 00:00:00:01:01:02
ip netns exec ls1p2 ip link set ls1p2 up
ip netns exec ls1p2 ip addr add 192.168.1.2/24 dev ls1p2
ip netns exec ls1p2 ip addr add 2001::2/64 dev ls1p2
ip netns exec ls1p2 ip route add default via 192.168.1.254 dev ls1p2
ip netns exec ls1p2 ip -6 route add default via 2001::a dev ls1p2
reproduced on ovn2.13-20.12.0-135:
[root@wsfd-advnetlab16 ~]# rpm -qa | grep -E "openvswitch2.15|ovn2.13"
ovn2.13-20.12.0-135.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn2.13-central-20.12.0-135.el8fdp.x86_64
openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn2.13-host-20.12.0-135.el8fdp.x86_64
[root@wsfd-advnetlab16 ~]# ovn-nbctl lr-nat-list lr1
TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat 172.17.1.11 192.168.1.1 00:00:00:00:ff:02 ls1p1
dnat_and_snat 172.17.1.12 192.168.1.2 00:00:00:00:ff:22 ls1p2
dnat_and_snat 7011::11 2001::1 00:00:00:00:ff:02 ls1p1
dnat_and_snat 7011::12 2001::2 00:00:00:00:ff:22 ls1p2
snat 172.17.1.41 192.168.1.0/24
snat 7011::41 2001::/64
[root@wsfd-advnetlab16 ~]# ip netns exec ls1p1 ping 172.17.1.102 -c 1
PING 172.17.1.102 (172.17.1.102) 56(84) bytes of data.
64 bytes from 172.17.1.102: icmp_seq=1 ttl=63 time=2.55 ms
--- 172.17.1.102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.554/2.554/2.554/0.000 ms
[root@wsfd-advnetlab17 bz1960096]# ip netns exec ls1p2 ping 172.17.1.102 -c 1
PING 172.17.1.102 (172.17.1.102) 56(84) bytes of data.
64 bytes from 172.17.1.102: icmp_seq=1 ttl=63 time=2.28 ms
--- 172.17.1.102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.284/2.284/2.284/0.000 ms
[root@wsfd-advnetlab16 bz1960096]# tcpdump -i ens1f1 -nnle icmp -v
dropped privs to tcpdump
tcpdump: listening on ens1f1, link-type EN10MB (Ethernet), capture size 262144 bytes
02:14:04.577481 00:00:00:00:ff:02 > f2:6f:3d:50:e2:70, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 22413, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.41 > 172.17.1.102: ICMP echo request, id 27568, seq 1, length 64
<=== src ip is the gateway ip (snat ip) for the packets coming out from ls1p1
02:14:04.578194 f2:6f:3d:50:e2:70 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 45145, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.41: ICMP echo reply, id 27568, seq 1, length 64
[root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp
dropped privs to tcpdump
tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes
02:13:58.720235 00:00:00:00:ff:22 > f2:6f:3d:50:e2:70, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 37466, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.12 > 172.17.1.102: ICMP echo request, id 26513, seq 1, length 64
<=== src ip is the FIP for ls1p2 for packet coming out from ls1p2
02:13:58.720291 f2:6f:3d:50:e2:70 > 00:00:00:00:ff:22, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 37528, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.12: ICMP echo reply, id 26513, seq 1, length 64
02:14:04.578050 00:00:00:00:ff:02 > f2:6f:3d:50:e2:70, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 22413, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.41 > 172.17.1.102: ICMP echo request, id 27568, seq 1, length 64
02:14:04.578093 f2:6f:3d:50:e2:70 > 00:00:00:00:00:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 45145, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.41: ICMP echo reply, id 27568, seq 1, length 64
the same for ipv6:
[root@wsfd-advnetlab17 bz1960096]# ip netns exec ls1p2 ping6 7011::102 -c 1
PING 7011::102(7011::102) 56 data bytes
64 bytes from 7011::102: icmp_seq=1 ttl=63 time=1.86 ms
--- 7011::102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.856/1.856/1.856/0.000 ms
[root@wsfd-advnetlab16 ~]# ip netns exec ls1p1 ping6 7011::102 -c 1
PING 7011::102(7011::102) 56 data bytes
64 bytes from 7011::102: icmp_seq=1 ttl=63 time=7.15 ms
--- 7011::102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 7.147/7.147/7.147/0.000 ms
[root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp6
dropped privs to tcpdump
tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes
02:14:56.558274 00:00:00:00:ff:22 > f2:6f:3d:50:e2:70, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xb159c, hlim 63, next-header ICMPv6 (58) payload length: 64) 7011::12 > 7011::102: [icmp6 sum ok] ICMP6, echo request, seq 1
02:14:56.558323 f2:6f:3d:50:e2:70 > 00:00:00:00:ff:22, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x5875e, hlim 64, next-header ICMPv6 (58) payload length: 64) 7011::102 > 7011::12: [icmp6 sum ok] ICMP6, echo reply, seq 1
02:15:14.929786 00:00:00:00:ff:02 > f2:6f:3d:50:e2:70, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xbe743, hlim 63, next-header ICMPv6 (58) payload length: 64) 7011::41 > 7011::102: [icmp6 sum ok] ICMP6, echo request, seq 1
02:15:14.929844 f2:6f:3d:50:e2:70 > 33:33:ff:00:00:41, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::102 > ff02::1:ff00:41: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 7011::41
source link-address option (1), length 8 (1): f2:6f:3d:50:e2:70
02:15:14.932349 00:00:00:00:00:02 > f2:6f:3d:50:e2:70, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::41 > 7011::102: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 7011::41, Flags [solicited, override]
destination link-address option (2), length 8 (1): 00:00:00:00:00:02
02:15:14.932378 f2:6f:3d:50:e2:70 > 00:00:00:00:00:02, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xc4eb9, hlim 64, next-header ICMPv6 (58) payload length: 64) 7011::102 > 7011::41: [icmp6 sum ok] ICMP6, echo reply, seq 1
Verified on ovn2.13-20.12.0-191:
[root@wsfd-advnetlab16 ~]# rpm -qa | grep -E "openvswitch2.15|ovn2.13"
ovn2.13-central-20.12.0-191.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn2.13-20.12.0-191.el8fdp.x86_64
ovn2.13-host-20.12.0-191.el8fdp.x86_64
openvswitch2.15-2.15.0-53.el8fdp.x86_64
[root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp
dropped privs to tcpdump
tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:21.244559 00:00:00:00:ff:22 > f2:6f:3d:50:e2:70, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 41840, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.12 > 172.17.1.102: ICMP echo request, id 27327, seq 1, length 64
02:20:21.244607 f2:6f:3d:50:e2:70 > 00:00:00:00:ff:22, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 60010, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.12: ICMP echo reply, id 27327, seq 1, length 64
02:20:25.624023 00:00:00:00:ff:02 > f2:6f:3d:50:e2:70, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 57207, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.11 > 172.17.1.102: ICMP echo request, id 28391, seq 1, length 64
<=== src ip if the FIP for ls1p1
02:20:25.624076 f2:6f:3d:50:e2:70 > 00:00:00:00:ff:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 40262, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.11: ICMP echo reply, id 28391, seq 1, length 64
[root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp6
dropped privs to tcpdump
tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes
02:20:34.261330 00:00:00:00:ff:22 > f2:6f:3d:50:e2:70, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xb159c, hlim 63, next-header ICMPv6 (58) payload length: 64) 7011::12 > 7011::102: [icmp6 sum ok] ICMP6, echo request, seq 1
02:20:34.261385 f2:6f:3d:50:e2:70 > 00:00:00:00:ff:22, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x5875e, hlim 64, next-header ICMPv6 (58) payload length: 64) 7011::102 > 7011::12: [icmp6 sum ok] ICMP6, echo reply, seq 1
02:20:37.723535 00:00:00:00:ff:02 > f2:6f:3d:50:e2:70, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xbe743, hlim 63, next-header ICMPv6 (58) payload length: 64) 7011::11 > 7011::102: [icmp6 sum ok] ICMP6, echo request, seq 1
02:20:37.723601 f2:6f:3d:50:e2:70 > 33:33:ff:00:00:11, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::102 > ff02::1:ff00:11: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 7011::11
source link-address option (1), length 8 (1): f2:6f:3d:50:e2:70
02:20:37.725389 00:00:00:00:ff:02 > f2:6f:3d:50:e2:70, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::11 > 7011::102: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 7011::11, Flags [solicited, override]
destination link-address option (2), length 8 (1): 00:00:00:00:ff:02
02:20:37.725410 f2:6f:3d:50:e2:70 > 00:00:00:00:ff:02, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x237de, hlim 64, next-header ICMPv6 (58) payload length: 64) 7011::102 > 7011::11: [icmp6 sum ok] ICMP6, echo reply, seq 1
also verified on ovn-2021-21.09.1-23:
[root@wsfd-advnetlab16 21.09.1-23]# rpm -qa | grep -E "openvswitch2.15|ovn-2021"
ovn-2021-host-21.09.1-23.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn-2021-central-21.09.1-23.el8fdp.x86_64
openvswitch2.15-2.15.0-53.el8fdp.x86_64
ovn-2021-21.09.1-23.el8fdp.x86_64
[root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp
dropped privs to tcpdump
tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes
02:25:13.343947 00:00:00:00:ff:22 > f2:65:1e:64:0f:77, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 28857, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.12 > 172.17.1.102: ICMP echo request, id 28947, seq 1, length 64
02:25:13.344013 f2:65:1e:64:0f:77 > 00:00:00:00:ff:22, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 42154, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.12: ICMP echo reply, id 28947, seq 1, length 64
02:25:18.295540 00:00:00:00:ff:02 > f2:65:1e:64:0f:77, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 59308, offset 0, flags [DF], proto ICMP (1), length 84)
172.17.1.11 > 172.17.1.102: ICMP echo request, id 30003, seq 1, length 64
<=== src ip is the FIP for ls1p1
02:25:18.296469 f2:65:1e:64:0f:77 > 00:00:00:00:ff:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 36702, offset 0, flags [none], proto ICMP (1), length 84)
172.17.1.102 > 172.17.1.11: ICMP echo reply, id 30003, seq 1, length 64
[root@wsfd-advnetlab17 ~]# ip netns exec ext2 tcpdump -i ext2 -nnle -v icmp6
dropped privs to tcpdump
tcpdump: listening on ext2, link-type EN10MB (Ethernet), capture size 262144 bytes
02:25:26.806524 00:00:00:00:ff:22 > 33:33:ff:00:01:02, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::200:ff:fe00:ff22 > ff02::1:ff00:102: [icmp6 sum ok]
ICMP6, neighbor solicitation, length 32, who has 7011::102
source link-address option (1), length 8 (1): 00:00:00:00:ff:22
02:25:26.806563 f2:65:1e:64:0f:77 > 00:00:00:00:ff:22, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::102 > fe80::200:ff:fe00:ff22: [icmp6 sum ok] ICMP6,
neighbor advertisement, length 32, tgt is 7011::102, Flags [solicited, override]
destination link-address option (2), length 8 (1): f2:65:1e:64:0f:77
02:25:26.808238 00:00:00:00:ff:22 > f2:65:1e:64:0f:77, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xb159c, hlim 63, next-header ICMPv6 (58) payload length: 64) 7011::12 > 7011::102: [icmp6 sum ok] ICMP6, echo request, seq 1
02:25:26.808312 f2:65:1e:64:0f:77 > 33:33:ff:00:00:12, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::102 > ff02::1:ff00:12: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 7011::12
source link-address option (1), length 8 (1): f2:65:1e:64:0f:77
02:25:26.809665 00:00:00:00:ff:22 > f2:65:1e:64:0f:77, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::12 > 7011::102: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 7011::12, Flags [solicited, override]
destination link-address option (2), length 8 (1): 00:00:00:00:ff:22
02:25:26.809701 f2:65:1e:64:0f:77 > 00:00:00:00:ff:22, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x5875e, hlim 64, next-header ICMPv6 (58) payload length: 64) 7011::102 > 7011::12: [icmp6 sum ok] ICMP6, echo reply, seq 1
02:25:30.258468 00:00:00:00:ff:02 > f2:65:1e:64:0f:77, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xbe743, hlim 63, next-header ICMPv6 (58) payload length: 64) 7011::11 > 7011::102: [icmp6 sum ok] ICMP6, echo request, seq 1
02:25:30.258518 f2:65:1e:64:0f:77 > 33:33:ff:00:00:11, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::102 > ff02::1:ff00:11: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 7011::11
source link-address option (1), length 8 (1): f2:65:1e:64:0f:77
02:25:30.260232 00:00:00:00:ff:02 > f2:65:1e:64:0f:77, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 7011::11 > 7011::102: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 7011::11, Flags [solicited, override]
destination link-address option (2), length 8 (1): 00:00:00:00:ff:02
02:25:30.260265 f2:65:1e:64:0f:77 > 00:00:00:00:ff:02, ethertype IPv6 (0x86dd), length 118: (flowlabel 0x237de, hlim 64, next-header ICMPv6 (58) payload length: 64) 7011::102 > 7011::11: [icmp6 sum ok] ICMP6, echo reply, seq 1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0049 |
Description of problem: When an instance has a floating ip and distributed_floating_ip is enabled, network traffic from instance to external network is supposed to have IP and MAC address of the floating ip. However, if an external gateway port of the router is located on the same node where the instance is running, packet is manipulated wrongly and gets IP address of external gateway and MAC address of the floating IP. On the other hand even in this situation any incoming packet to the floating IP uses IP and MAC address of the floating IP. This results in duplicated association of floating ip MAC address (with external gateway IP and floating ip IP), and causes flapping. ~~~ $ openstack port list --long | egrep -e floating -e gateway | 96a3eab1-e860-4171-9f3b-18fba0f026f4 | | fa:16:3e:05:06:44 | ip_address='10.0.0.210', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | N/A | None | network:floatingip | | | d8866825-dba8-407a-bfc5-30a3f24d86b3 | | fa:16:3e:5a:24:55 | ip_address='10.0.0.242', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | N/A | None | network:floatingip | | | e441a1d9-4a13-42a5-be39-b683636a3086 | | fa:16:3e:08:fa:4a | ip_address='10.0.0.215', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | ACTIVE | None | network:router_gateway | | $ openstack server show testinstance002 +-------------------------------------+---------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+---------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-1.redhat.local | | OS-EXT-SRV-ATTR:hostname | testinstance002 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.redhat.local | | OS-EXT-SRV-ATTR:instance_name | instance-00000005 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-2cuw6ukr | | OS-EXT-SRV-ATTR:root_device_name | /dev/vda | | OS-EXT-SRV-ATTR:user_data | None | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2021-04-15T07:18:24.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | private=192.168.10.216, 10.0.0.210 | | config_drive | | | created | 2021-04-15T07:18:11Z | | description | None | | flavor | disk='1', ephemeral='0', extra_specs.hw_rng:allowed='True', original_name='m1.nano', ram='128', swap='0', vcpus='1' | | hostId | e8ed5b800e6f03df8bbfd667f1d078a5fff553e24c2bb0931e48005b | | host_status | UP | | id | 3a821599-d8db-43c3-9d8f-9ec9d84fc2b7 | | image | cirros-0.4.0-x86_64-disk.img (ad38b060-abdc-4570-a46a-1c799fb46898) | | key_name | None | | locked | False | | locked_reason | None | | name | testinstance002 | | progress | 0 | | project_id | 4c9a7610e1b043be9ba5fcb530a964ad | | properties | | | security_groups | name='icmp' | | server_groups | [] | | status | ACTIVE | | tags | [] | | trusted_image_certificates | None | | updated | 2021-04-15T07:18:24Z | | user_id | 492083d2deef4aaaae5dbd0cc4e3df19 | | volumes_attached | | +-------------------------------------+---------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port show e441a1d9-4a13-42a5-be39-b683636a3086 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | compute-1.redhat.local | | binding_profile | | | binding_vif_details | port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2021-04-15T06:55:07Z | | data_plane_status | None | | description | | | device_id | 6f7b40aa-c1ec-4e07-972d-a71af103db7d | | device_owner | network:router_gateway | | dns_assignment | fqdn='host-10-0-0-215.openstacklocal.', hostname='host-10-0-0-215', ip_address='10.0.0.215' | | | fqdn='host-2620-52-0-13b8--1000-33.openstacklocal.', hostname='host-2620-52-0-13b8--1000-33', ip_address='2620:52:0:13b8::1000:33' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.0.215', subnet_id='6661e943-1789-439e-b957-65d93748fa8c' | | | ip_address='2620:52:0:13b8::1000:33', subnet_id='20ed52a9-1788-4ad8-8e4a-8d0d40e6eb73' | | id | e441a1d9-4a13-42a5-be39-b683636a3086 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='4c9a7610e1b043be9ba5fcb530a964ad', project.name='admin', region_name='regionOne', zone= | | mac_address | fa:16:3e:08:fa:4a | | name | | | network_id | 08012876-fb02-4f3e-9000-40810c433c3c | | port_security_enabled | False | | project_id | | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 372 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2021-05-06T05:59:48Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ~~~ /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ~~~ [ovn] ... enable_distributed_floating_ip=True ~~~ The following tcpdump was captured while ping 8.8.8.8 from the instance. I see all packets use fa:16:3e:05:06:44 (floating ip MAC) and 10.0.0.215 (gateway port IP) ~~~ 05:59:14.817170 fa:16:3e:05:06:44 > 52:54:00:14:90:f3, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 48427, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.0.215 > 8.8.8.8: ICMP echo request, id 41731, seq 23, length 64 05:59:14.824942 52:54:00:14:90:f3 > fa:16:3e:08:fa:4a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 112, id 0, offset 0, flags [none], proto ICMP (1), length 84) 8.8.8.8 > 10.0.0.215: ICMP echo reply, id 41731, seq 23, length 64 05:59:15.817602 fa:16:3e:05:06:44 > 52:54:00:14:90:f3, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 48600, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.0.215 > 8.8.8.8: ICMP echo request, id 41731, seq 24, length 64 05:59:15.825358 52:54:00:14:90:f3 > fa:16:3e:08:fa:4a, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 112, id 0, offset 0, flags [none], proto ICMP (1), length 84) 8.8.8.8 > 10.0.0.215: ICMP echo reply, id 41731, seq 24, length 64 ~~~ Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Deploy overcloud with ovn and NeutronEnableDVR: true 2. Disable gateway on Controller nodes (*) $ sudo ovs-vsctl remove open . external_ids ovn-cms-options 3. Enable gateway on Compute nodes (*) $ sudo ovs-vsctl set open . external_ids:ovn-cms-options=enable-chassis-as-gw 4. Create networks and router 5. Create an instance and assign floating ip to the instance 6. ping external system from the instance and observe packets in external interface (*) These steps are required to schedule gateway port in compute nodes. Actual results: The traffic has gateway IP used Expected results: The traffic has floating ip IP used Additional info: This issue was observed in a DCN deployment in ovn. Because each site has different physical networks, router gateways were scheduled on computes in a remote site instead of controller nodes.