+++ This bug was initially created as a clone of Bug #1643902 +++ +++ This bug was initially created as a clone of Bug #1643900 +++ MAC_Binding entries are not updated when Floating IPs are added so when a new FIP is assigned an old gw IP address, it'll be unreachable as wrong MAC address is used. Details at: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-October/047604.html So far we workarounded it in networking-ovn but this should have proper fix in core OVN side.
Patch is merged upstream in master branch: https://github.com/openvswitch/ovs/commit/81e928526b8a9393b90785fb0a9c82d79570ef84 Is it possible to get a backport to 2.9? I guess this would be ds only
If not possible, we will just apply the workaround on networking-ovn side
(In reply to Daniel Alvarez Sanchez from comment #4) > Patch is merged upstream in master branch: > https://github.com/openvswitch/ovs/commit/ > 81e928526b8a9393b90785fb0a9c82d79570ef84 > Is it possible to get a backport to 2.9? I guess this would be ds only Done. It's available in 2.9.0-89.el7fdn
According to our records, this should be resolved by openvswitch-2.9.0-83.el7fdp.1. This build is available now.
Tested on puddle: OpenStack/13.0-RHEL-7/2019-01-22.1 rpm: openvswitch-2.9.0-83.el7fdp.1.x86_64 The issue still occurs. Steps that I did before the issue re-occurred: Started from clean system. 1. Created internal network internal_A, subnet_A (192.168.3.0/24), router_A. Connected router_A to the external network. Connected internal network internal_A to the router. Created a VM and connected to the internal_A network. Attached an floating IP to the VM. 2. Created internal network internal_B, subnet_B (192.168.4.0/24), router_B. Connected router_B to the external network. Connected internal network internal_B to the router. Created a VM and connected to the internal_B network. Attached an floating IP to the VM. 3. On this stage I was able to ping VMs with floating IPs from external network and from other VM with floating IP (OK) 4. I deleted one of floating IPs. Then I re-created it again (same ip address, used command 'openstack floating ip create --floating-ip-address 10.0.0.220 nova' , 'nova' is external network name) 5. Attached the FIP to the other VM on other internal network, connected to other router. New mac_binding entry in north db was created (OK). Ping and ssh to this VM via FIP worked, however I noticed that arp table on hypevisor was not updated with new mac for 10.0.0.220. 6. Created internal network internal_C, subnet_C (192.168.5.0/24), router_C. Connected router_C to the external network. Connected internal network internal_C to the router. Created a VM and connected to the internal_C network. 7. Deleted FIP 10.0.0.20 and recreated it again (see step 4). Then attached it to the VM connected to internal_C network and router_C. 8. Connected to VM on internal_B network (via internal interface in namespace on compute node, the VM has FIP 10.0.0.212), tried to ping 10.0.0.220 - no replies (BUG) This time MAC address in arp table was updated properly (OK) [root@titan09 ~]# arp -n | grep 220 10.0.0.220 ether fa:16:3e:1a:b9:41 C external However MAC_Binding table does not have entry for new mac (BAD) _uuid : 983f17a8-39eb-42ef-81f7-014af443a257 datapath : e430caa6-45d6-4c89-b4ab-5f6fdd64421a ip : "10.0.0.220" logical_port : "lrp-1ad704fd-fe3d-45dc-8da0-de2c13bace3e" mac : "fa:16:3e:47:cc:79" _uuid : ed5dd9c1-81c1-4a8f-838f-329935403508 datapath : 3835e5d6-3cca-4568-a1d6-559d0de53095 ip : "10.0.0.212" logical_port : "lrp-140de39c-b91c-4bce-9f80-42c0f9bdb646" mac : "fa:16:3e:7a:cc:c2" _uuid : 80cef92d-7f6d-4787-90ef-43c03765b605 datapath : e430caa6-45d6-4c89-b4ab-5f6fdd64421a ip : "10.0.0.1" logical_port : "lrp-1ad704fd-fe3d-45dc-8da0-de2c13bace3e" mac : "52:54:00:90:eb:a9" _uuid : 44f147b2-9033-4fa8-bf80-7df7b8c57e60 datapath : d59102ca-a542-48bc-9d6a-d815865e9c48 ip : "10.0.0.1" logical_port : "lrp-95a230dc-355e-4a0f-a480-4aec7876df5a" mac : "52:54:00:90:eb:a9" _uuid : 23336992-adf1-4461-864f-b44589e78615 datapath : 3835e5d6-3cca-4568-a1d6-559d0de53095 ip : "10.0.0.220" logical_port : "lrp-140de39c-b91c-4bce-9f80-42c0f9bdb646" mac : "fa:16:3e:7a:cc:c2" _uuid : 0a3c46b6-1e46-42d0-abe8-5d12796015da datapath : e430caa6-45d6-4c89-b4ab-5f6fdd64421a ip : "10.0.0.211" logical_port : "lrp-1ad704fd-fe3d-45dc-8da0-de2c13bace3e" mac : "fa:16:3e:47:cc:79" _uuid : 34f08edb-7c1e-4783-88c4-93f8aa6f2205 datapath : 3835e5d6-3cca-4568-a1d6-559d0de53095 ip : "10.0.0.1" logical_port : "lrp-140de39c-b91c-4bce-9f80-42c0f9bdb646" mac : "52:54:00:90:eb:a9" 'ovn-nbctl show' output switch 695bcff4-7363-414f-b220-26da7f8cc65a (neutron-41728a0a-ee96-4924-b186-2026f6de66fe) (aka nova) port 140de39c-b91c-4bce-9f80-42c0f9bdb646 type: router router-port: lrp-140de39c-b91c-4bce-9f80-42c0f9bdb646 port 1ad704fd-fe3d-45dc-8da0-de2c13bace3e type: router router-port: lrp-1ad704fd-fe3d-45dc-8da0-de2c13bace3e port 95a230dc-355e-4a0f-a480-4aec7876df5a type: router router-port: lrp-95a230dc-355e-4a0f-a480-4aec7876df5a port bb87562e-72b3-433b-9d35-5510753058ec type: localport addresses: ["fa:16:3e:98:3e:66"] port provnet-41728a0a-ee96-4924-b186-2026f6de66fe type: localnet addresses: ["unknown"] switch 0ead1979-c739-443e-b218-7f8f1ad8439c (neutron-08dad4fc-6343-48be-8d36-638eb2fb429e) (aka internal_B) port d61578bd-b68c-4c60-99f3-de715680bbf3 addresses: ["fa:16:3e:49:8d:19 192.168.4.17"] port 6023cc1b-1b73-4154-8990-4861eadf080c addresses: ["fa:16:3e:13:c3:b0 192.168.4.8"] port 7116fe78-eb78-41dc-933b-cef29df90594 type: localport addresses: ["fa:16:3e:73:4b:40 192.168.4.2"] port 28bf9b5d-35a7-471c-a41b-20455b384091 type: router router-port: lrp-28bf9b5d-35a7-471c-a41b-20455b384091 port 82f37936-50cf-429b-aac2-77cb1f5da84a addresses: ["fa:16:3e:a5:e8:a2 192.168.4.28"] switch 44b34927-080d-4611-861a-d78ddf3fed6f (neutron-3512997c-85f4-4abb-aa2a-72f477f6f210) (aka internal_C) port b1d736ab-ff6d-4420-b29c-b8f2902e4d3b type: router router-port: lrp-b1d736ab-ff6d-4420-b29c-b8f2902e4d3b port 613aa235-b953-4c00-9c28-d9f937c10db4 addresses: ["fa:16:3e:b8:44:03 192.168.5.8"] port 7dac22b5-fb59-4250-81bb-a7de9abe9572 type: localport addresses: ["fa:16:3e:3c:42:4c 192.168.5.2"] switch 9fc44ed7-90a4-4315-bcba-ac59a848d80f (neutron-aea40033-582b-4d0b-b144-05e678d6e9cf) (aka internal_A) port 8f7b4921-17d9-4aaf-8ab6-2ca789abaed1 addresses: ["fa:16:3e:03:08:a8 192.168.3.5"] port c65a2216-2a95-46a8-9a39-c9e935912f00 addresses: ["fa:16:3e:e5:01:04 192.168.3.27"] port 12fb654b-8a0d-4821-8de7-3214b19ba04f type: router router-port: lrp-12fb654b-8a0d-4821-8de7-3214b19ba04f port de34f213-677e-4804-a6a1-4f1ae97a5722 type: localport addresses: ["fa:16:3e:5e:2f:34 192.168.3.2"] router 8e2e6395-3d9a-4b34-a66d-75a3d2814702 (neutron-d1b6a1cc-5470-41a0-b3a8-970e6cffa128) (aka routerC) port lrp-95a230dc-355e-4a0f-a480-4aec7876df5a mac: "fa:16:3e:1a:b9:41" networks: ["10.0.0.217/24"] gateway chassis: [5ddb95b8-4303-45c5-8cf9-cbacffeafc08 3c0ba0c0-b214-40a5-8823-268fbc3aa80e 5d86147f-c008-4029-a7cb-0aa3c1cf5793] port lrp-b1d736ab-ff6d-4420-b29c-b8f2902e4d3b mac: "fa:16:3e:f8:d1:f8" networks: ["192.168.5.1/24"] nat 84d15a9d-250c-429d-918c-8aa2b06ce12d external ip: "10.0.0.220" logical ip: "192.168.5.8" type: "dnat_and_snat" nat c010110e-a10a-4815-9a20-3f961a523538 external ip: "10.0.0.217" logical ip: "192.168.5.0/24" type: "snat" router 4dd82062-886b-4aa0-84b8-4b3b8ea4744c (neutron-4face940-f57c-4bb7-b3fe-ea96040cd31b) (aka routerB) port lrp-140de39c-b91c-4bce-9f80-42c0f9bdb646 mac: "fa:16:3e:47:cc:79" networks: ["10.0.0.223/24"] gateway chassis: [5d86147f-c008-4029-a7cb-0aa3c1cf5793 5ddb95b8-4303-45c5-8cf9-cbacffeafc08 3c0ba0c0-b214-40a5-8823-268fbc3aa80e] port lrp-28bf9b5d-35a7-471c-a41b-20455b384091 mac: "fa:16:3e:3a:7e:fe" networks: ["192.168.4.1/24"] nat a68563bd-93c5-485b-90e8-c1e258cb7a82 external ip: "10.0.0.211" logical ip: "192.168.4.8" type: "dnat_and_snat" nat d09e2f63-9d3d-4bea-9cdd-82a1e8fc4af3 external ip: "10.0.0.223" logical ip: "192.168.4.0/24" type: "snat" router 81db4a46-0024-4d39-abb1-2693d21975a4 (neutron-679cead8-e326-4dc1-9c9d-ecfde38b3274) (aka routerA) port lrp-1ad704fd-fe3d-45dc-8da0-de2c13bace3e mac: "fa:16:3e:7a:cc:c2" networks: ["10.0.0.224/24"] gateway chassis: [3c0ba0c0-b214-40a5-8823-268fbc3aa80e 5ddb95b8-4303-45c5-8cf9-cbacffeafc08 5d86147f-c008-4029-a7cb-0aa3c1cf5793] port lrp-12fb654b-8a0d-4821-8de7-3214b19ba04f mac: "fa:16:3e:68:be:79" networks: ["192.168.3.1/24"] nat 1e984589-ca55-4c04-9485-98eae836bb5e external ip: "10.0.0.212" logical ip: "192.168.3.5" type: "dnat_and_snat" nat 365d2cdd-31c2-4efb-8652-3c785e551081 external ip: "10.0.0.224" logical ip: "192.168.3.0/24" type: "snat"
Verified on 13.0-RHEL-7/2019-03-01.1 with openvswitch-2.9.0-97.el7fdp.x86_64 and python-networking-ovn-4.0.3-3.el7ost.noarch Used the following scenario: 1. Created internal network internal_A, subnet_A (192.168.1.0/24), router_A. Connected router_A to the external network. Connected internal network internal_A to the router. 2. Created VM1 and connected to the internal_A network. Attached a floating IP (10.0.0.177) to the VM1. 3. Created VM2 and connected to the internal_A network. Attached a floating IP (10.0.0.173) to the VM2. 4. Verified that connectivity works in all directions between floating IPs and external network gateway. 5. Created internal network internal_B, subnet_B (192.168.2.0/24), router_B. Connected router_B to the external network. Connected internal network internal_B to the router. 6. Deleted one of the floating IP that was in use (10.0.0.177) and recreated it again with another uuid but same IP. Used command 'openstack floating ip create --floating-ip-address 10.0.0.177 external' 7. Created VM3 and connected to the internal_B network. Attached the floating IP 10.0.0.177 to it. Verified that arp table on the external network is updated with a new MAC for 10.0.0.177 8. Verified that VM3 is reachable using the assigned floating IP. Verified that connectivity between all servers works using floating IPs. 9. Repeated steps 6-9 with second floating IP and VM4 connected to internal_B network. All worked as expected.