Description of problem: This is an issue with DVR routers unable to resolve arp for vip addresses created as allowed-address-pairs when the instance that's hosting the vip is on a different compute node and the router namespace that's trying to resolve arp for the vip address is on a different compute node Same works fine when trying from the router namespace created on the compute node where the instance is running that's hosting the vip Here are the details from my lab: There are 2 vxlan networks: +++ (overcloud) [stack@undercloud16 ~]$ neutron net-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+------+----------------------------------+----------------------------------------------------+ | id | name | tenant_id | subnets | +--------------------------------------+------+----------------------------------+----------------------------------------------------+ | 1c6adde3-3959-4ae5-be0c-36cb71a41e1f | net1 | d20a046f5c9140a78993c1e0dd3e6b58 | 12b24308-c016-4617-a13d-c590586d382b 172.16.2.0/24 | | 5381a89b-4403-4f09-8f0e-2ed69fe9f9c2 | net2 | d20a046f5c9140a78993c1e0dd3e6b58 | 41fc52da-0b3c-4bef-a54f-85ba9c237827 192.24.5.0/24 | +--------------------------------------+------+----------------------------------+----------------------------------------------------+ (overcloud) [stack@undercloud16 ~]$ neutron subnet-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+---------+----------------------------------+---------------+------------------------------------------------+ | id | name | tenant_id | cidr | allocation_pools | +--------------------------------------+---------+----------------------------------+---------------+------------------------------------------------+ | 12b24308-c016-4617-a13d-c590586d382b | subnet1 | d20a046f5c9140a78993c1e0dd3e6b58 | 172.16.2.0/24 | {"start": "172.16.2.2", "end": "172.16.2.254"} | | 41fc52da-0b3c-4bef-a54f-85ba9c237827 | subnet2 | d20a046f5c9140a78993c1e0dd3e6b58 | 192.24.5.0/24 | {"start": "192.24.5.2", "end": "192.24.5.254"} | +--------------------------------------+---------+----------------------------------+---------------+------------------------------------------------+ +++ Both of these networks are connected to the same router: +++ (overcloud) [stack@undercloud16 ~]$ neutron router-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+---------+----------------------------------+-----------------------+-------------+-------+ | id | name | tenant_id | external_gateway_info | distributed | ha | +--------------------------------------+---------+----------------------------------+-----------------------+-------------+-------+ | 9ea43f9e-841a-4f31-a00b-bb0b021556a8 | router1 | d20a046f5c9140a78993c1e0dd3e6b58 | null | True | False | +--------------------------------------+---------+----------------------------------+-----------------------+-------------+-------+ (overcloud) [stack@undercloud16 ~]$ neutron router-port-list 9ea43f9e-841a-4f31-a00b-bb0b021556a8 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+------+----------------------------------+-------------------+-----------------------------------------------------------------------------------+ | id | name | tenant_id | mac_address | fixed_ips | +--------------------------------------+------+----------------------------------+-------------------+-----------------------------------------------------------------------------------+ | 43aa2c7a-2e9a-422e-a255-562cd5ab6a08 | | d20a046f5c9140a78993c1e0dd3e6b58 | fa:16:3e:49:b7:ad | {"subnet_id": "12b24308-c016-4617-a13d-c590586d382b", "ip_address": "172.16.2.1"} | | c6c6de08-aefb-4aab-b8cf-0da6285ac58b | | d20a046f5c9140a78993c1e0dd3e6b58 | fa:16:3e:5b:92:51 | {"subnet_id": "41fc52da-0b3c-4bef-a54f-85ba9c237827", "ip_address": "192.24.5.1"} | +--------------------------------------+------+----------------------------------+-------------------+-----------------------------------------------------------------------------------+ +++ I have 2 compute nodes in this lab and 1 controller node: +++ (undercloud) [stack@undercloud16 ~]$ nova list +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ | 0b5acc55-3f55-488e-a01d-d3c6b5404c85 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.24.24 | | a4c38228-a4ff-410d-8ba1-ddad1f66ff93 | overcloud-novacompute-0 | ACTIVE | - | Running | ctlplane=192.168.24.7 | | dba27db5-33a7-4f40-b9bf-9b04f760e1d3 | overcloud-novacompute-1 | ACTIVE | - | Running | ctlplane=192.168.24.13 | +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ (undercloud) [stack@undercloud16 ~]$ +++ then I created 2 instances on the 2 networks: +++ (overcloud) [stack@undercloud16 ~]$ nova show 8ea2104f-e274-49e1-bc75-12d586b6da81 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-novacompute-0.site1.redhat.local | | OS-EXT-SRV-ATTR:hostname | test1 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-0.site1.redhat.local | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-jknt9fzp | | OS-EXT-SRV-ATTR:root_device_name | /dev/vda | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2021-04-22T08:27:20.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2021-04-22T08:27:04Z | | description | - | | flavor:disk | 10 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | m1.small | | flavor:ram | 1024 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | 12cde44bad72a26453736eb64754bb7442275fff90de022d79fbf5b4 | | host_status | UP | | id | 8ea2104f-e274-49e1-bc75-12d586b6da81 | | image | rhel7 (58b0637f-5c8f-41a9-b0ef-62d7bb6ec5ac) | | key_name | key1 | | locked | False | | locked_reason | - | | metadata | {} | | name | test1 | | net1 network | 172.16.2.229 | (overcloud) [stack@undercloud16 ~]$ nova show a47fc4a2-68bc-4b4d-81ac-27d65b130860 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | overcloud-novacompute-1.site1.redhat.local | | OS-EXT-SRV-ATTR:hostname | test2 | | OS-EXT-SRV-ATTR:hypervisor_hostname | overcloud-novacompute-1.site1.redhat.local | | OS-EXT-SRV-ATTR:instance_name | instance-00000004 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-a0ia46y5 | | OS-EXT-SRV-ATTR:root_device_name | /dev/vda | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2021-04-22T08:28:59.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2021-04-22T08:28:45Z | | description | - | | flavor:disk | 10 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | m1.small | | flavor:ram | 1024 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | 0d667a202bfefd3eb31a1edf62b64c4a2b8c6ca3ff084f17638caad0 | | host_status | UP | | id | a47fc4a2-68bc-4b4d-81ac-27d65b130860 | | image | rhel7 (58b0637f-5c8f-41a9-b0ef-62d7bb6ec5ac) | | key_name | key1 | | locked | False | | locked_reason | - | | metadata | {} | | name | test2 | | net2 network | 192.24.5.184 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | secgroup1 | +++ Note that both the instances are on 2 different compute nodes here are the security group rules for the security group for the instances: +++ (overcloud) [stack@undercloud16 ~]$ openstack security group rule list secgroup1 +--------------------------------------+-------------+-----------+-----------+------------+-----------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group | +--------------------------------------+-------------+-----------+-----------+------------+-----------------------+ | 5021ef1c-ba5e-4411-8583-5f012b3e72be | None | IPv4 | 0.0.0.0/0 | | None | | 50a8de1a-000c-44a8-b39a-03fe4e7abb2d | None | IPv6 | ::/0 | | None | | 87452278-fe04-4a33-9260-9b6175cda680 | icmp | IPv4 | 0.0.0.0/0 | | None | | e78fd417-fca6-438b-a6c8-18852acae20b | tcp | IPv4 | 0.0.0.0/0 | 22:22 | None | +--------------------------------------+-------------+-----------+-----------+------------+-----------------------+ +++ then I created a vip port with the same security group: +++ (overcloud) [stack@undercloud16 ~]$ neutron port-create --fixed-ip subnet_id=12b24308-c016-4617-a13d-c590586d382b,ip_address=172.16.2.25 1c6adde3-3959-4ae5-be0c-36cb71a41e1f neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Created a new port: +-----------------------+------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | | | binding:profile | {} | | binding:vif_details | {} | | binding:vif_type | unbound | | binding:vnic_type | normal | | created_at | 2021-04-22T08:36:10Z | | description | | | device_id | | | device_owner | | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "12b24308-c016-4617-a13d-c590586d382b", "ip_address": "172.16.2.25"} | | id | c8f99338-4b15-407c-bfe1-3de559b5b8ac | | ip_allocation | immediate | | mac_address | fa:16:3e:7f:71:66 | | name | | | network_id | 1c6adde3-3959-4ae5-be0c-36cb71a41e1f | | port_security_enabled | True | | project_id | d20a046f5c9140a78993c1e0dd3e6b58 | | qos_policy_id | | | resource_request | | | revision_number | 1 | | security_groups | 0de5a2ad-1521-4324-a66f-da1d61c34abc | | status | DOWN | | tags | | | tenant_id | d20a046f5c9140a78993c1e0dd3e6b58 | | updated_at | 2021-04-22T08:36:11Z | +-----------------------+------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud16 ~]$ neutron port-update --security-group 21ebe018-314b-4c85-88ef-1465b4f25653 c8f99338-4b15-407c-bfe1-3de559b5b8ac neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Updated port: c8f99338-4b15-407c-bfe1-3de559b5b8ac (overcloud) [stack@undercloud16 ~]$ neutron port-show c8f99338-4b15-407c-bfe1-3de559b5b8ac neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +-----------------------+------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | | | binding:profile | {} | | binding:vif_details | {} | | binding:vif_type | unbound | | binding:vnic_type | normal | | created_at | 2021-04-22T08:36:10Z | | description | | | device_id | | | device_owner | | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "12b24308-c016-4617-a13d-c590586d382b", "ip_address": "172.16.2.25"} | | id | c8f99338-4b15-407c-bfe1-3de559b5b8ac | | ip_allocation | immediate | | mac_address | fa:16:3e:7f:71:66 | | name | | | network_id | 1c6adde3-3959-4ae5-be0c-36cb71a41e1f | | port_security_enabled | True | | project_id | d20a046f5c9140a78993c1e0dd3e6b58 | | qos_policy_id | | | resource_request | | | revision_number | 2 | | security_groups | 21ebe018-314b-4c85-88ef-1465b4f25653 | | status | DOWN | | tags | | | tenant_id | d20a046f5c9140a78993c1e0dd3e6b58 | | updated_at | 2021-04-22T08:37:21Z | +-----------------------+------------------------------------------------------------------------------------+ +++ then I added this port as a allowed-address-pair on the interface for the test1 instance: +++ (overcloud) [stack@undercloud16 ~]$ nova list +--------------------------------------+-------+--------+------------+-------------+-------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+-------------------+ | 8ea2104f-e274-49e1-bc75-12d586b6da81 | test1 | ACTIVE | - | Running | net1=172.16.2.229 | | a47fc4a2-68bc-4b4d-81ac-27d65b130860 | test2 | ACTIVE | - | Running | net2=192.24.5.184 | +--------------------------------------+-------+--------+------------+-------------+-------------------+ (overcloud) [stack@undercloud16 ~]$ nova interface-list 8ea2104f-e274-49e1-bc75-12d586b6da81 +------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+ | Port State | Port ID | Net ID | IP addresses | MAC Addr | Tag | +------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+ | ACTIVE | d667e73e-9ff3-4821-9858-8093cef4e142 | 1c6adde3-3959-4ae5-be0c-36cb71a41e1f | 172.16.2.229 | fa:16:3e:5d:0a:70 | - | +------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+ (overcloud) [stack@undercloud16 ~]$ neutron port-update --allowed-address-pair ip_address=172.16.2.25 d667e73e-9ff3-4821-9858-8093cef4e142 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Updated port: d667e73e-9ff3-4821-9858-8093cef4e142 (overcloud) [stack@undercloud16 ~]$ neutron port-show d667e73e-9ff3-4821-9858-8093cef4e142 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +-----------------------+--------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | {"mac_address": "fa:16:3e:5d:0a:70", "ip_address": "172.16.2.25"} | | binding:host_id | overcloud-novacompute-0.site1.redhat.local | | binding:profile | {} | | binding:vif_details | {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": "system", "bridge_name": "br-int"} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2021-04-22T08:27:11Z | | description | | | device_id | 8ea2104f-e274-49e1-bc75-12d586b6da81 | | device_owner | compute:nova | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "12b24308-c016-4617-a13d-c590586d382b", "ip_address": "172.16.2.229"} | | id | d667e73e-9ff3-4821-9858-8093cef4e142 | | ip_allocation | immediate | | mac_address | fa:16:3e:5d:0a:70 | | name | | | network_id | 1c6adde3-3959-4ae5-be0c-36cb71a41e1f | | port_security_enabled | True | | project_id | d20a046f5c9140a78993c1e0dd3e6b58 | | qos_policy_id | | | resource_request | | | revision_number | 5 | | security_groups | 21ebe018-314b-4c85-88ef-1465b4f25653 | | status | ACTIVE | | tags | | | tenant_id | d20a046f5c9140a78993c1e0dd3e6b58 | | updated_at | 2021-04-22T08:38:45Z | +-----------------------+--------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud16 ~]$ +++ then inside the test1 instance; I set the vip address on the port: +++ [root@test1 ~]# nmcli conn show NAME UUID TYPE DEVICE System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet eth0 [root@test1 ~]# nmcli conn mod "System eth0" +ipv4.address 172.16.2.25/24 [root@test1 ~]# nmcli conn up "System eth0" Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2) [root@test1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:5d:0a:70 brd ff:ff:ff:ff:ff:ff inet 172.16.2.229/24 brd 172.16.2.255 scope global noprefixroute dynamic eth0 valid_lft 86399sec preferred_lft 86399sec inet 172.16.2.25/24 brd 172.16.2.255 scope global secondary noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe5d:a70/64 scope link tentative valid_lft forever preferred_lft forever [root@test1 ~]# +++ After that I checked from the router namespace on compute-0 where the instance which hosts the vip is running; the router namesapce on this compute node is able to resolve the arp: +++ [root@overcloud-novacompute-0 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 ip neigh 172.16.2.2 dev qr-43aa2c7a-2e lladdr fa:16:3e:5a:82:8d PERMANENT 172.16.2.229 dev qr-43aa2c7a-2e lladdr fa:16:3e:5d:0a:70 PERMANENT 192.24.5.2 dev qr-c6c6de08-ae lladdr fa:16:3e:4b:80:d5 PERMANENT 172.16.2.25 dev qr-43aa2c7a-2e lladdr fa:16:3e:5d:0a:70 STALE fe80::f816:3eff:fe35:723f dev qr-c6c6de08-ae lladdr fa:16:3e:35:72:3f STALE fe80::f816:3eff:fe5d:a70 dev qr-43aa2c7a-2e lladdr fa:16:3e:5d:0a:70 STALE [root@overcloud-novacompute-0 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 ping 172.16.2.25 PING 172.16.2.25 (172.16.2.25) 56(84) bytes of data. 64 bytes from 172.16.2.25: icmp_seq=1 ttl=64 time=2.85 ms ^C --- 172.16.2.25 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 2.848/2.848/2.848/0.000 ms [root@overcloud-novacompute-0 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 22: qr-43aa2c7a-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:49:b7:ad brd ff:ff:ff:ff:ff:ff inet 172.16.2.1/24 brd 172.16.2.255 scope global qr-43aa2c7a-2e valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe49:b7ad/64 scope link valid_lft forever preferred_lft forever 24: qr-c6c6de08-ae: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:5b:92:51 brd ff:ff:ff:ff:ff:ff inet 192.24.5.1/24 brd 192.24.5.255 scope global qr-c6c6de08-ae valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe5b:9251/64 scope link valid_lft forever preferred_lft forever [root@overcloud-novacompute-0 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 arping -I qr-43aa2c7a-2e 172.16.2.25 ARPING 172.16.2.25 from 172.16.2.1 qr-43aa2c7a-2e Unicast reply from 172.16.2.25 [FA:16:3E:5D:0A:70] 2.511ms Unicast reply from 172.16.2.25 [FA:16:3E:5D:0A:70] 1.413ms ^CSent 2 probes (1 broadcast(s)) Received 2 response(s) [root@overcloud-novacompute-0 ~]# +++ As this is a dvr router; the same namespace exists on compute-1 and the router on compute-1 is not able to resolve the arp for the vip address 172.16.2.25: +++ [root@overcloud-novacompute-1 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 11: qr-43aa2c7a-2e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:49:b7:ad brd ff:ff:ff:ff:ff:ff inet 172.16.2.1/24 brd 172.16.2.255 scope global qr-43aa2c7a-2e valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe49:b7ad/64 scope link valid_lft forever preferred_lft forever 13: qr-c6c6de08-ae: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:5b:92:51 brd ff:ff:ff:ff:ff:ff inet 192.24.5.1/24 brd 192.24.5.255 scope global qr-c6c6de08-ae valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe5b:9251/64 scope link valid_lft forever preferred_lft forever [root@overcloud-novacompute-1 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 arping -I qr-43aa2c7a-2e 172.16.2.25 ARPING 172.16.2.25 from 172.16.2.1 qr-43aa2c7a-2e ^CSent 6 probes (6 broadcast(s)) Received 0 response(s) [root@overcloud-novacompute-1 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 ip neigh 172.16.2.25 dev qr-43aa2c7a-2e lladdr fa:16:3e:5d:0a:70 STALE 192.24.5.184 dev qr-c6c6de08-ae lladdr fa:16:3e:35:72:3f STALE 172.16.2.142 dev qr-43aa2c7a-2e lladdr fa:16:3e:8f:af:5e PERMANENT 172.16.2.2 dev qr-43aa2c7a-2e lladdr fa:16:3e:5a:82:8d PERMANENT 192.24.5.223 dev qr-c6c6de08-ae lladdr fa:16:3e:2b:0b:b6 PERMANENT 192.24.5.2 dev qr-c6c6de08-ae lladdr fa:16:3e:4b:80:d5 PERMANENT fe80::f816:3eff:fe35:723f dev qr-c6c6de08-ae lladdr fa:16:3e:35:72:3f STALE fe80::f816:3eff:fe2b:bb6 dev qr-c6c6de08-ae lladdr fa:16:3e:2b:0b:b6 STALE fe80::f816:3eff:fe5d:a70 dev qr-43aa2c7a-2e lladdr fa:16:3e:5d:0a:70 STALE fe80::f816:3eff:fe8f:af5e dev qr-43aa2c7a-2e lladdr fa:16:3e:8f:af:5e STALE [root@overcloud-novacompute-1 ~]# [root@overcloud-novacompute-1 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 arp -n Address HWtype HWaddress Flags Mask Iface 172.16.2.25 (incomplete) qr-43aa2c7a-2e 192.24.5.184 ether fa:16:3e:35:72:3f C qr-c6c6de08-ae 172.16.2.142 ether fa:16:3e:8f:af:5e CM qr-43aa2c7a-2e 172.16.2.2 ether fa:16:3e:5a:82:8d CM qr-43aa2c7a-2e 192.24.5.223 ether fa:16:3e:2b:0b:b6 CM qr-c6c6de08-ae 192.24.5.2 ether fa:16:3e:4b:80:d5 CM qr-c6c6de08-ae [root@overcloud-novacompute-1 ~]# +++ As a result test2 instance running on compute-1 isn't able to reach the vip address: +++ [root@overcloud-controller-0 ~]# ip netns exec qdhcp-5381a89b-4403-4f09-8f0e-2ed69fe9f9c2 ssh -i /home/heat-admin/id_rsa cloud-user.5.184 The authenticity of host '192.24.5.184 (192.24.5.184)' can't be established. ECDSA key fingerprint is SHA256:dQ5mhGV8XiXAqP7D9LjGIYWUw8pS8YRoVUXYmUjdDeM. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.24.5.184' (ECDSA) to the list of known hosts. [cloud-user@test2 ~]$ sudo su - [root@test2 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:35:72:3f brd ff:ff:ff:ff:ff:ff inet 192.24.5.184/24 brd 192.24.5.255 scope global noprefixroute dynamic eth0 valid_lft 85177sec preferred_lft 85177sec inet6 fe80::f816:3eff:fe35:723f/64 scope link valid_lft forever preferred_lft forever [root@test2 ~]# ping 172.16.2.25 PING 172.16.2.25 (172.16.2.25) 56(84) bytes of data. From 192.24.5.1 icmp_seq=1 Destination Host Unreachable From 192.24.5.1 icmp_seq=2 Destination Host Unreachable From 192.24.5.1 icmp_seq=3 Destination Host Unreachable From 192.24.5.1 icmp_seq=4 Destination Host Unreachable ^C --- 172.16.2.25 ping statistics --- 7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6001ms pipe 4 [root@test2 ~]# +++ Just to check if this was an issue with only compute-0; I created another instance on the same network from where the vip is created and this was created on compute-1 and this instance was able to reach the vip: +++ (overcloud) [stack@undercloud16 ~]$ nova boot --flavor 8562c82c-12ec-499e-9451-51868ff6b137 --image 58b0637f-5c8f-41a9-b0ef-62d7bb6ec5ac --nic net-id=1c6adde3-3959-4ae5-be0c-36cb71a41e1f --security-groups 21ebe018-314b-4c85-88ef-1465b4f25653 --key-name key1 --availability-zone nova:overcloud-novacompute-1.site1.redhat.local test_beta --poll +--------------------------------------+----------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | test-beta | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-nle8u2bq | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | KPfpt8iKNhto | | config_drive | | | created | 2021-04-22T08:55:13Z | | description | - | | flavor:disk | 10 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | m1.small | | flavor:ram | 1024 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | | | host_status | | | id | 5eac4f72-fd73-4c59-94c9-032680923c85 | | image | rhel7 (58b0637f-5c8f-41a9-b0ef-62d7bb6ec5ac) | | key_name | key1 | | locked | False | | locked_reason | - | | metadata | {} | | name | test_beta | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | 21ebe018-314b-4c85-88ef-1465b4f25653 | | server_groups | [] | | status | BUILD | | tags | [] | | tenant_id | d20a046f5c9140a78993c1e0dd3e6b58 | | trusted_image_certificates | - | | updated | 2021-04-22T08:55:13Z | | user_id | cee4e8b3f4e644409b665c8f6b76afee | +--------------------------------------+----------------------------------------------+ Server building... 100% complete Finished (overcloud) [stack@undercloud16 ~]$ nova list +--------------------------------------+-----------+--------+------------+-------------+-------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-----------+--------+------------+-------------+-------------------+ | 8ea2104f-e274-49e1-bc75-12d586b6da81 | test1 | ACTIVE | - | Running | net1=172.16.2.229 | | a47fc4a2-68bc-4b4d-81ac-27d65b130860 | test2 | ACTIVE | - | Running | net2=192.24.5.184 | | 5eac4f72-fd73-4c59-94c9-032680923c85 | test_beta | ACTIVE | - | Running | net1=172.16.2.94 | +--------------------------------------+-----------+--------+------------+-------------+-------------------+ [root@test-beta ~]# ping 172.16.2.25 PING 172.16.2.25 (172.16.2.25) 56(84) bytes of data. 64 bytes from 172.16.2.25: icmp_seq=1 ttl=64 time=8.43 ms 64 bytes from 172.16.2.25: icmp_seq=2 ttl=64 time=1.91 ms 64 bytes from 172.16.2.25: icmp_seq=3 ttl=64 time=2.53 ms ^C --- 172.16.2.25 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.913/4.293/8.435/2.940 ms [root@test-beta ~]# +++ The only workaround thus far is to create a static entry in the arp table of the router on compute-1 and then test2 can reach the vip: +++ [root@overcloud-novacompute-1 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 arp -s 172.16.2.25 fa:16:3e:5d:0a:70 [root@overcloud-novacompute-1 ~]# ip netns exec qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 arp -n Address HWtype HWaddress Flags Mask Iface 172.16.2.2 ether fa:16:3e:5a:82:8d CM qr-43aa2c7a-2e 172.16.2.142 ether fa:16:3e:8f:af:5e CM qr-43aa2c7a-2e 172.16.2.25 ether fa:16:3e:5d:0a:70 CM qr-43aa2c7a-2e 192.24.5.2 ether fa:16:3e:4b:80:d5 CM qr-c6c6de08-ae 172.16.2.94 ether fa:16:3e:90:cb:bd C qr-43aa2c7a-2e 192.24.5.223 ether fa:16:3e:2b:0b:b6 CM qr-c6c6de08-ae 192.24.5.184 ether fa:16:3e:35:72:3f C qr-c6c6de08-ae [root@overcloud-novacompute-1 ~]# [root@test2 ~]# ping 172.16.2.25 PING 172.16.2.25 (172.16.2.25) 56(84) bytes of data. 64 bytes from 172.16.2.25: icmp_seq=1 ttl=63 time=5.55 ms 64 bytes from 172.16.2.25: icmp_seq=2 ttl=63 time=2.05 ms 64 bytes from 172.16.2.25: icmp_seq=3 ttl=63 time=2.97 ms ^C --- 172.16.2.25 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 2.055/3.528/5.555/1.482 ms [root@test2 ~]# +++ this workaround is however not scalable and we would notice the same issue in case the vip moves from this instance to a different instance Version-Release number of selected component (if applicable): [root@overcloud-controller-0 ~]# podman ps | grep -i neutron a1044dc7e0ec undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.1.5 /usr/sbin/dnsmasq... 34 minutes ago Up 34 minutes ago neutron-dnsmasq-qdhcp-5a1bb7a4-bbf5-4f79-9a4c-934fe526e0a0 481920da5c1c undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.1.5 /bin/bash -c HAPR... 6 hours ago Up 6 hours ago neutron-haproxy-qrouter-9ea43f9e-841a-4f31-a00b-bb0b021556a8 594ca39706b0 undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.1.5 /usr/sbin/dnsmasq... 6 hours ago Up 6 hours ago neutron-dnsmasq-qdhcp-5381a89b-4403-4f09-8f0e-2ed69fe9f9c2 ebbe3e86d813 undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.1.5 /usr/sbin/dnsmasq... 6 hours ago Up 6 hours ago neutron-dnsmasq-qdhcp-1c6adde3-3959-4ae5-be0c-36cb71a41e1f 37a596822ebd undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-openvswitch-agent:16.1.5 kolla_start 6 hours ago Up 6 hours ago neutron_ovs_agent fa2733131fb6 undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-metadata-agent:16.1.5 kolla_start 6 hours ago Up 6 hours ago neutron_metadata_agent fb3ee5c02bef undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-l3-agent:16.1.5 kolla_start 6 hours ago Up 6 hours ago neutron_l3_agent 96581c7e2f6b undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.1.5 kolla_start 6 hours ago Up 6 hours ago neutron_dhcp d7a5f8c1aa4a undercloud16.ctlplane.site1.redhat.local:8787/rhosp-rhel8/openstack-neutron-server:16.1.5 kolla_start 6 hours ago Up 6 hours ago neutron_api [root@overcloud-controller-0 ~]# podman exec -it 37a596822ebd rpm -qa | grep -i neutron python3-neutron-lib-1.29.1-1.20201113130108.4ef4b71.el8ost.noarch openstack-neutron-openvswitch-15.2.1-1.20201114025041.el8ost.noarch puppet-neutron-15.5.1-1.20201113235652.el8ost.noarch openstack-neutron-common-15.2.1-1.20201114025041.el8ost.noarch openstack-neutron-15.2.1-1.20201114025041.el8ost.noarch python3-neutron-dynamic-routing-15.0.1-1.20200821103647.56de1c4.el8ost.noarch python3-neutronclient-6.14.1-1.20201113231846.a09e824.el8ost.noarch python3-neutron-15.2.1-1.20201114025041.el8ost.noarch openstack-neutron-ml2-15.2.1-1.20201114025041.el8ost.noarch [root@overcloud-controller-0 ~]# How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I also tried enabling arp_responder in openvswitch_agent.ini and restart neutron_ovs_agent container with the systemd unit and still I don't see any arp_tpa flows for the vip address on any of the compute nodes: +++ [root@overcloud-novacompute-0 ~]# grep -i arp_responder /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini arp_responder=True [root@overcloud-novacompute-0 ~]# ovs-ofctl dump-flows br-int cookie=0x26b98293dee12182, duration=5586.025s, table=0, n_packets=0, n_bytes=0, priority=65535,vlan_tci=0x0fff/0x1fff actions=drop cookie=0x26b98293dee12182, duration=5582.127s, table=0, n_packets=1, n_bytes=86, priority=10,icmp6,in_port="qvod667e73e-9f",icmp_type=136 actions=resubmit(,24) cookie=0x26b98293dee12182, duration=5582.119s, table=0, n_packets=4348, n_bytes=182616, priority=10,arp,in_port="qvod667e73e-9f" actions=resubmit(,24) cookie=0x26b98293dee12182, duration=5585.415s, table=0, n_packets=48, n_bytes=3300, priority=2,in_port="int-br-ex" actions=drop cookie=0x26b98293dee12182, duration=5582.134s, table=0, n_packets=982, n_bytes=94531, priority=9,in_port="qvod667e73e-9f" actions=resubmit(,25) cookie=0x26b98293dee12182, duration=5585.359s, table=0, n_packets=0, n_bytes=0, priority=5,in_port="int-br-ex",dl_dst=fa:16:3f:4c:a9:45 actions=resubmit(,4) cookie=0x26b98293dee12182, duration=5585.356s, table=0, n_packets=33, n_bytes=1386, priority=5,in_port="patch-tun",dl_dst=fa:16:3f:4c:a9:45 actions=resubmit(,3) cookie=0x26b98293dee12182, duration=5585.370s, table=0, n_packets=0, n_bytes=0, priority=4,in_port="int-br-ex",dl_src=fa:16:3f:20:fb:22 actions=resubmit(,2) cookie=0x26b98293dee12182, duration=5585.364s, table=0, n_packets=2097, n_bytes=175490, priority=2,in_port="patch-tun",dl_src=fa:16:3f:20:fb:22 actions=resubmit(,1) cookie=0x26b98293dee12182, duration=5585.353s, table=0, n_packets=0, n_bytes=0, priority=4,in_port="int-br-ex",dl_src=fa:16:3f:c9:c7:90 actions=resubmit(,2) cookie=0x26b98293dee12182, duration=5585.348s, table=0, n_packets=0, n_bytes=0, priority=2,in_port="patch-tun",dl_src=fa:16:3f:c9:c7:90 actions=resubmit(,1) cookie=0x26b98293dee12182, duration=5586.031s, table=0, n_packets=9556, n_bytes=626951, priority=0 actions=resubmit(,60) cookie=0x26b98293dee12182, duration=5582.598s, table=1, n_packets=10, n_bytes=812, priority=20,dl_vlan=3,dl_dst=fa:16:3e:5d:0a:70 actions=mod_dl_src:fa:16:3e:49:b7:ad,resubmit(,60) cookie=0x26b98293dee12182, duration=5585.418s, table=1, n_packets=413, n_bytes=17346, priority=1 actions=drop cookie=0x26b98293dee12182, duration=5585.417s, table=2, n_packets=0, n_bytes=0, priority=1 actions=drop cookie=0x8dd12943fa3fed19, duration=22329.509s, table=3, n_packets=32, n_bytes=1344, priority=5,dl_vlan=1,dl_dst=fa:16:3f:4c:a9:45 actions=mod_dl_dst:fa:16:3e:5b:92:51,strip_vlan,output:4 cookie=0x8dd12943fa3fed19, duration=22329.401s, table=3, n_packets=0, n_bytes=0, priority=5,dl_vlan=2,dl_dst=fa:16:3f:4c:a9:45 actions=mod_dl_dst:fa:16:3e:49:b7:ad,strip_vlan,output:3 cookie=0x26b98293dee12182, duration=5582.621s, table=3, n_packets=0, n_bytes=0, priority=5,dl_vlan=3,dl_dst=fa:16:3f:4c:a9:45 actions=mod_dl_dst:fa:16:3e:49:b7:ad,strip_vlan,output:"qr-43aa2c7a-2e" cookie=0x26b98293dee12182, duration=5582.361s, table=3, n_packets=1, n_bytes=42, priority=5,dl_vlan=4,dl_dst=fa:16:3f:4c:a9:45 actions=mod_dl_dst:fa:16:3e:5b:92:51,strip_vlan,output:"qr-c6c6de08-ae" cookie=0x26b98293dee12182, duration=5585.420s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x26b98293dee12182, duration=5582.130s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port="qvod667e73e-9f",icmp_type=136,nd_target=fe80::f816:3eff:fe5d:a70 actions=resubmit(,60) cookie=0x26b98293dee12182, duration=5582.125s, table=24, n_packets=16, n_bytes=672, priority=2,arp,in_port="qvod667e73e-9f",arp_spa=172.16.2.229 actions=resubmit(,25) cookie=0x26b98293dee12182, duration=5582.122s, table=24, n_packets=4321, n_bytes=181482, priority=2,arp,in_port="qvod667e73e-9f",arp_spa=172.16.2.25 actions=resubmit(,25) cookie=0x26b98293dee12182, duration=5586.027s, table=24, n_packets=1, n_bytes=86, priority=0 actions=drop cookie=0x26b98293dee12182, duration=5582.140s, table=25, n_packets=5328, n_bytes=276967, priority=2,in_port="qvod667e73e-9f",dl_src=fa:16:3e:5d:0a:70 actions=resubmit(,60) cookie=0x26b98293dee12182, duration=5582.595s, table=60, n_packets=951, n_bytes=85273, priority=20,dl_vlan=3,dl_dst=fa:16:3e:5d:0a:70 actions=strip_vlan,output:"qvod667e73e-9f" cookie=0x26b98293dee12182, duration=5586.029s, table=60, n_packets=15801, n_bytes=1149208, priority=3 actions=NORMAL cookie=0x26b98293dee12182, duration=5586.022s, table=61, n_packets=0, n_bytes=0, priority=3 actions=NORMAL +++
*** Bug 1985035 has been marked as a duplicate of this bug. ***