Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1818695

Summary: l3 DVR ARP population gets incorrect MAC address
Product: Red Hat OpenStack Reporter: Brendan Shephard <bshephar>
Component: openstack-neutronAssignee: Slawek Kaplonski <skaplons>
Status: CLOSED ERRATA QA Contact: Alex Katz <akatz>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: amuller, bcafarel, ccamposr, chris.smart, chrisw, jrfuller, michjohn, mparkins, nsatsia, peli, pmannidi, scohen, skaplons, sputhenp
Target Milestone: z12Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: All   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-12.1.1-17.el7ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-24 11:53:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
files used in hotfix verification none

Description Brendan Shephard 2020-03-30 06:33:54 UTC 
Description of problem:
When using DVR, if the qrouter is unable to resolve a MAC address using arping, it tries to get the MAC from the ports defined in Neutron. This works provided we are not using allowed-address-pairs, such as with Octavia. In this situation, the MAC address will be populated incorrectly since the IP isn't assigned to an interface with a unique MAC address, rather it is created as a secondary IP on the Amphora interface

Version-Release number of selected component (if applicable):
openstack-neutron-12.1.1-6.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create 2 VM's with allowed-address-pairs and 1 extra Port that matches the allowed-address-pair
2. Configure the extra port IP as a secondary IP on one of the VM's
3. Restart neutron_l3_agent

Actual results:
The qrouter namespace will now have the incorrect MAC address as it is getting the MAC address from the port, rather than checking the allowed-address section:
https://opendev.org/openstack/neutron/src/branch/stable/queens/neutron/agent/l3/dvr_local_router.py#L282-L298

Expected results:
Ideally, we should rely on the results of the actual network and actual ARP responses rather than trying to preempt them according to data in the Neutron DB. However, a potential solution might be to also check the allowed-address section of the port, if something exists, use that MAC address, else default to the MAC address of the port.

Additional info:
This issue happens by default with DVR and Octavia. When the Load Balancer is first created, it isn't an issue. But after restarting l3_agent, it will have the incorrect MAC address using the FIP port that is created to reserve the IP address. 


[root@compute-0 ~]# ip netns exec qrouter-d4b46d21-951a-4ee6-b06e-1408f0e8d7b8 arp -a                                                                                                                  [66/1980]
? (192.168.99.17) at fa:16:3e:13:83:6b [ether] PERM on qr-a74c4fc7-77
? (172.30.128.1) at <incomplete> on qr-acd0b7fe-50
? (192.168.99.36) at fa:16:3e:a7:50:8b [ether] PERM on qr-a74c4fc7-77
gateway (172.30.128.9) at fa:16:3e:ff:ae:7c [ether] PERM on qr-acd0b7fe-50
? (192.168.99.26) at fa:16:3e:f8:41:5d [ether] PERM on qr-a74c4fc7-77
? (192.168.99.8) at fa:16:3e:f8:41:5d [ether] PERM on qr-a74c4fc7-77
gateway (169.254.95.213) at a2:b5:36:f1:c0:fe [ether] PERM on rfp-d4b46d21-9
? (172.30.0.1) at fa:16:3e:a3:f3:76 [ether] PERM on qr-acd0b7fe-50     <<<<<---- This one is incorrect

We can see why it's incorrect in the Amphora VM:
In the Amphora VM:
[root@amphora-d5e70510-9325-4047-9067-449b1397ba0a ~]# ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8946 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:19:69:fe brd ff:ff:ff:ff:ff:ff
    inet 172.30.128.11/16 brd 172.30.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 172.30.0.1/16 brd 172.30.255.255 scope global secondary eth1:0
       valid_lft forever preferred_lft forever

We can fix it like this:
Delete the ARP entry:
[root@hp-comp96-0 ~]# ip netns exec qrouter-d4b46d21-951a-4ee6-b06e-1408f0e8d7b8 arp -d 172.30.0.1

Add the correct ARP entry:
[root@hp-comp96-0 ~]# ip netns exec qrouter-d4b46d21-951a-4ee6-b06e-1408f0e8d7b8 arp -s 172.30.0.1 fa:16:3e:19:69:fe

Now the PING is working:
From 192.168.99.1 icmp_seq=193 Destination Host Unreachable
From 192.168.99.1 icmp_seq=194 Destination Host Unreachable
From 192.168.99.1 icmp_seq=195 Destination Host Unreachable
From 192.168.99.1 icmp_seq=199 Destination Host Unreachable
64 bytes from 172.30.0.1: icmp_seq=203 ttl=63 time=688 ms
64 bytes from 172.30.0.1: icmp_seq=202 ttl=63 time=1687 ms
64 bytes from 172.30.0.1: icmp_seq=204 ttl=63 time=0.502 ms
64 bytes from 172.30.0.1: icmp_seq=205 ttl=63 time=0.577 ms
64 bytes from 172.30.0.1: icmp_seq=206 ttl=63 time=0.514 ms

We can see how we end up in this situation by looking at the ports:
So that is coming from here:
(overcloud) [stack@director deployment]$ neutron port-list | grep 172.30.0.1
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
| 346891a4-a231-41cf-b08b-77a71c61ebe2 | octavia-lb-9798d544-8b26-421c-ac76-dd761bc4e654                               | f7c3ec7446184de5a61edfdbbe78bf4a | fa:16:3e:a3:f3:76 | {"subnet_id": "0c3faffb-2412-49ed-9972-2ffd6a3ed77a", "ip_address": "172.30.0.1"}      |


(overcloud) [stack@director deployment]$ neutron port-list | grep 172.30.128.11
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
| f9fe7cbe-5cc7-41e7-a415-b46c71b5f51f | octavia-lb-vrrp-d5e70510-9325-4047-9067-449b1397ba0a                          | 76958bf7fb484ceaad8e6760fa738715 | fa:16:3e:19:69:fe | {"subnet_id": "0c3faffb-2412-49ed-9972-2ffd6a3ed77a", "ip_address": "172.30.128.11"}   |
Comment 7 Brendan Shephard 2020-03-31 21:22:21 UTC 
The temporary solution to this problem has been to add the following:

https://opendev.org/openstack/neutron/src/branch/master/neutron/common/utils.py#L198-L199

    181 def get_dvr_allowed_address_pair_device_owners():
    182     """Return device_owner names for allowed_addr_pair ports serviced by DVR
    183 
    184     This just returns the device owners that are used by the
    185     allowed_address_pair ports. Right now only the device_owners shown
    186     below are used by the allowed_address_pair ports.
    187     Later if other device owners are used for allowed_address_pairs those
    188     device_owners should be added to the list below.
    189     """
    190     # TODO(Swami): Convert these methods to constants.
    191     # Add the constants variable to the neutron-lib
    192     return [n_const.DEVICE_OWNER_LOADBALANCER,
    193             n_const.DEVICE_OWNER_LOADBALANCERV2,
    194             "Octavia"]

Note: We added line 194. This was patched in the neutron_l3_agent container and the container was restarted. After which, we deleted the ARP entry, added the correct one to the qrouter namespace and restarted l3_agent again. We noted that it no longer impacted the correct ARP entry.
Comment 14 Candido Campos 2020-05-05 10:41:57 UTC 
Created attachment 1685160 [details]
files used in hotfix verification

files used in hotfix verification
Comment 28 errata-xmlrpc 2020-06-24 11:53:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2724