Description of problem: Using the steps mentioned in [1], Earlier in Ovs based deployment we were successfully using "Highly available VIPs on OpenStack VMs with VRRP". We can successfully ping floating ip in ovs based deployment. In OVN based deployment, We cannot ping floating ip associated with VIP. Looking to ways, workaround, fix to make this work. [1] https://blog.codecentric.de/en/2016/11/highly-available-vips-openstack-vms-vrrp/ It seems like some flow rules are missing for VIP. ~~~ [root@overcloud-controller-0 ~]# ovn-nbctl --db=tcp:172.168.125.21:6641 lr-nat-list e41593b7-84e3-4cab-8daf-1d924b1d3e14 TYPE EXTERNAL_IP LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat 10.10.10.102 26.26.26.4 ----> VIP dnat_and_snat 10.10.10.108 26.26.26.7 fa:16:3e:70:e2:d7 70a77356-5e56-4616-914a-bff2bee94d61 dnat_and_snat 10.10.10.112 26.26.26.12 fa:16:3e:f4:39:2f 6d35f1bc-6e34-47e6-bf00-1146d6bef08e snat 10.10.10.103 26.26.26.0/24 ~~~ Version-Release number of selected component (if applicable): OSP 13 + OVN How reproducible: Steps to Reproduce: 1. Perform steps as mentioned in [1] ~~~ https://blog.codecentric.de/en/2016/11/highly-available-vips-openstack-vms-vrrp/ ~~~ * We have created instances and vip in below network:- (overcloud) [root@dell-r440-20 ~]# openstack network list +--------------------------------------+----------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+----------+--------------------------------------+ | db0dc2cf-1ed8-4e3e-a806-b5f1cb7c6f11 | private | 7b08899f-d866-418c-8080-b2e84efa4310 | +--------------------------------------+----------+--------------------------------------+ * Following ports were created:- ~~~ (overcloud) [root@dell-r440-20 ~]# openstack port list | grep -i port | 356f1ffd-69b1-456b-a48e-9beb35e6c98e | vip-port | fa:16:3e:31:14:2b | ip_address='26.26.26.4', subnet_id='7b08899f-d866-418c-8080-b2e84efa4310' | DOWN | | 6d35f1bc-6e34-47e6-bf00-1146d6bef08e | vm1-port | fa:16:3e:b1:4e:cb | ip_address='26.26.26.12', subnet_id='7b08899f-d866-418c-8080-b2e84efa4310' | ACTIVE | | 70a77356-5e56-4616-914a-bff2bee94d61 | vm2-port | fa:16:3e:55:13:74 | ip_address='26.26.26.7', subnet_id='7b08899f-d866-418c-8080-b2e84efa4310' | ACTIVE | ~~~ * We associated private IP to floating ips ~~~ (overcloud) [root@dell-r440-20 ~]# openstack floating ip list +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ | 1d61e370-3745-4dd3-924a-40cf111ca232 | 10.10.10.112 | 26.26.26.12 | 6d35f1bc-6e34-47e6-bf00-1146d6bef08e | 377ea1cf-e9d8-47d5-b374-58e3c81f28aa | 2839e80d82f643afa981dea0db522b91 | | 48127265-8332-4fbe-bb8c-65c8a78db2bb | 10.10.10.108 | 26.26.26.7 | 70a77356-5e56-4616-914a-bff2bee94d61 | 377ea1cf-e9d8-47d5-b374-58e3c81f28aa | 2839e80d82f643afa981dea0db522b91 | | ab123cd9-17f1-49ea-a542-fa77a52a91c7 | 10.10.10.102 | 26.26.26.4 | 356f1ffd-69b1-456b-a48e-9beb35e6c98e | 377ea1cf-e9d8-47d5-b374-58e3c81f28aa | 2839e80d82f643afa981dea0db522b91 | +--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+ * Create necessary security group rule and assign it to vm ports * Create RHEL VMs using the ports. (overcloud) [root@dell-r440-20 ~]# openstack server list --long +--------------------------------------+------+--------+------------+-------------+-----------------------------------+------------+--------------------------------------+-------------+--------------------------------------+-------------------+-----------------------------------------+------------+ | ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Flavor Name | Flavor ID | Availability Zone | Host | Properties | +--------------------------------------+------+--------+------------+-------------+-----------------------------------+------------+--------------------------------------+-------------+--------------------------------------+-------------------+-----------------------------------------+------------+ | ceaaab83-4d77-45e2-bf88-ddadfb08512d | vm2 | ACTIVE | None | Running | private=26.26.26.7, 10.10.10.108 | rhel | 3f80e1ed-a0b2-430f-b9b4-7664281da465 | rhel | 2107f813-5ab4-481e-bc5d-4500ee9e02d7 | nova | overcloud-novacompute-dvr-1.localdomain | | | 2d2ba1fe-ce5f-4bcd-bb62-f1f51e27b54f | vm1 | ACTIVE | None | Running | private=26.26.26.12, 10.10.10.112 | rhel | 3f80e1ed-a0b2-430f-b9b4-7664281da465 | rhel | 2107f813-5ab4-481e-bc5d-4500ee9e02d7 | nova | overcloud-novacompute-dvr-0.localdomain | | +--------------------------------------+------+--------+------------+-------------+-----------------------------------+------------+--------------------------------------+-------------+--------------------------------------+-------------------+-----------------------------------------+------------+ * VM1 currently have the vip:- ~~~ [root@vm1 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:b1:4e:cb brd ff:ff:ff:ff:ff:ff inet 26.26.26.12/24 brd 26.26.26.255 scope global noprefixroute dynamic eth0 ------> VM1 port ip valid_lft 32403sec preferred_lft 32403sec inet 26.26.26.4/32 scope global eth0 -----------------> VIP valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feb1:4ecb/64 scope link valid_lft forever preferred_lft forever conf:- [root@vm1 ~]# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script "killall -0 haproxy" # check the haproxy process interval 2 # every 2 seconds weight 2 # add 2 points if OK } vrrp_instance VI_1 { interface eth0 # interface to monitor state MASTER # MASTER on haproxy, BACKUP on haproxy2 virtual_router_id 51 priority 101 # 101 on haproxy, 100 on haproxy2 virtual_ipaddress { 26.26.26.4 # virtual ip address } track_script { chk_haproxy } } ~~~ * IP details and keepalive configuration of second vm:- ~~~ [root@vm2 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:55:13:74 brd ff:ff:ff:ff:ff:ff inet 26.26.26.7/24 brd 26.26.26.255 scope global noprefixroute dynamic eth0 valid_lft 31510sec preferred_lft 31510sec inet6 fe80::f816:3eff:fe55:1374/64 scope link valid_lft forever preferred_lft forever [root@vm2 ~]# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script "killall -0 haproxy" # check the haproxy process interval 2 # every 2 seconds weight 2 # add 2 points if OK } vrrp_instance VI_1 { interface eth0 # interface to monitor state BACKUP # MASTER on haproxy, BACKUP on haproxy2 virtual_router_id 51 priority 100 # 101 on haproxy, 100 on haproxy2 virtual_ipaddress { 26.26.26.4 # virtual ip address } track_script { chk_haproxy } } ~~~ * Interesting, VM1 and VIP private ip is reachable from ovnmetadata namespace from compute0 ~~~ [root@overcloud-novacompute-dvr-0 ~]# ip netns exec ovnmeta-b6611c71-9d30-424e-8dc5-a75fdcf6b30a ping 26.26.26.12 PING 26.26.26.12 (26.26.26.12) 56(84) bytes of data. 64 bytes from 26.26.26.12: icmp_seq=1 ttl=64 time=1.19 ms ^C --- 26.26.26.12 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.199/1.199/1.199/0.000 ms [root@overcloud-novacompute-dvr-0 ~]# ip netns exec ovnmeta-b6611c71-9d30-424e-8dc5-a75fdcf6b30a ping 26.26.26.4 --------------> VIP IP reachable from ovnmeta namespace PING 26.26.26.4 (26.26.26.4) 56(84) bytes of data. 64 bytes from 26.26.26.4: icmp_seq=1 ttl=64 time=0.991 ms 64 bytes from 26.26.26.4: icmp_seq=2 ttl=64 time=0.913 ms ^C --- 26.26.26.4 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.913/0.952/0.991/0.039 ms * Also, VM2 IP is reachable from ovnmetadata namespace from compute1. [root@overcloud-novacompute-dvr-1 ~]# ip netns exec ovnmeta-b6611c71-9d30-424e-8dc5-a75fdcf6b30a ping 26.26.26.7 PING 26.26.26.7 (26.26.26.7) 56(84) bytes of data. 64 bytes from 26.26.26.7: icmp_seq=1 ttl=64 time=1.01 ms ^C --- 26.26.26.7 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.016/1.016/1.016/0.000 ms ~~~ * Floating ip associated with VM is pingable ~~~ (overcloud) [root@dell-r440-20 ~]# ping 10.10.10.108 PING 10.10.10.108 (10.10.10.108) 56(84) bytes of data. 64 bytes from 10.10.10.108: icmp_seq=1 ttl=63 time=1.64 ms ^C --- 10.10.10.108 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.645/1.645/1.645/0.000 ms (overcloud) [root@dell-r440-20 ~]# ping 10.10.10.112 PING 10.10.10.112 (10.10.10.112) 56(84) bytes of data. 64 bytes from 10.10.10.112: icmp_seq=1 ttl=63 time=1.80 ms ^C --- 10.10.10.112 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.800/1.800/1.800/0.000 ms ~~~ * But, Floating IP associated with vip is not pingable. ~~~ (overcloud) [root@dell-r440-20 ~]# ping 10.10.10.102 PING 10.10.10.102 (10.10.10.102) 56(84) bytes of data. ^C --- 10.10.10.102 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 999ms ~~~ It seems like some flow rules are missing for VIP. ~~~ [root@overcloud-controller-0 ~]# ovn-nbctl --db=tcp:172.168.125.21:6641 lr-nat-list e41593b7-84e3-4cab-8daf-1d924b1d3e14 TYPE EXTERNAL_IP LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat 10.10.10.102 26.26.26.4 ----> VIP dnat_and_snat 10.10.10.108 26.26.26.7 fa:16:3e:70:e2:d7 70a77356-5e56-4616-914a-bff2bee94d61 dnat_and_snat 10.10.10.112 26.26.26.12 fa:16:3e:f4:39:2f 6d35f1bc-6e34-47e6-bf00-1146d6bef08e snat 10.10.10.103 26.26.26.0/24 ~~~ Actual results: Floating ip is not reachable Expected results: Floating IP should be reachable as It used to work ovs based environment. Additional info:
I verified that failover works properly with OSP13 + openvswitch-2.9.0-110.el7fdp.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1744