Bug 1707241

Summary: Unable to use "Highly available VIPs on OpenStack VMs with VRRP" in OVN based deployment
Product: Red Hat OpenStack Reporter: Sandeep Yadav <sandyada>
Component: python-networking-ovnAssignee: Maciej Józefczyk <mjozefcz>
Status: CLOSED ERRATA QA Contact: Roman Safronov <rsafrono>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: amuller, apevec, dalvarez, ekuris, lhh, lmartins, majopela, mgeary, mjozefcz, scohen, twilson
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: python-networking-ovn-4.0.3-7.el7ost Doc Type: Bug Fix
Doc Text:
Previously, Address Resolution Protocol (ARP) failed for High Availability floating IP with Virtual Router Redundancy Protocol (VRRP) deployment. With this update, floating IP on Virtual IP works correctly.
 Story Points: ---
Clone Of:
: 1714119 1722790 (view as bug list) Environment:
Last Closed: 2019-07-10 13:02:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1722790    
Bug Blocks:    

Description Sandeep Yadav 2019-05-07 06:53:08 UTC 
Description of problem:

Using the steps mentioned in [1], Earlier in Ovs based deployment we were successfully using "Highly available VIPs on OpenStack VMs with VRRP". We can successfully ping floating ip in ovs based deployment.

In OVN based deployment, We cannot ping floating ip associated with VIP.

Looking to ways, workaround, fix to make this work.


[1] https://blog.codecentric.de/en/2016/11/highly-available-vips-openstack-vms-vrrp/ 



It seems like some flow rules are missing for VIP.

	~~~
	[root@overcloud-controller-0 ~]#  ovn-nbctl --db=tcp:172.168.125.21:6641 lr-nat-list e41593b7-84e3-4cab-8daf-1d924b1d3e14
	TYPE             EXTERNAL_IP        LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
	dnat_and_snat    10.10.10.102       26.26.26.4                                                                       ----> VIP
	dnat_and_snat    10.10.10.108       26.26.26.7            fa:16:3e:70:e2:d7    70a77356-5e56-4616-914a-bff2bee94d61
	dnat_and_snat    10.10.10.112       26.26.26.12           fa:16:3e:f4:39:2f    6d35f1bc-6e34-47e6-bf00-1146d6bef08e
	snat             10.10.10.103       26.26.26.0/24
	~~~


Version-Release number of selected component (if applicable):

OSP 13 + OVN

How reproducible:


Steps to Reproduce:


1. Perform steps as mentioned in [1]

	~~~
	https://blog.codecentric.de/en/2016/11/highly-available-vips-openstack-vms-vrrp/
	~~~


* We have created instances and vip in below network:-

	(overcloud) [root@dell-r440-20 ~]# openstack network list
	+--------------------------------------+----------+--------------------------------------+
	| ID                                   | Name     | Subnets                              |
	+--------------------------------------+----------+--------------------------------------+
	| db0dc2cf-1ed8-4e3e-a806-b5f1cb7c6f11 | private  | 7b08899f-d866-418c-8080-b2e84efa4310 |
	+--------------------------------------+----------+--------------------------------------+


* Following ports were created:-
	~~~
	(overcloud) [root@dell-r440-20 ~]# openstack port list | grep -i port
	| 356f1ffd-69b1-456b-a48e-9beb35e6c98e | vip-port | fa:16:3e:31:14:2b | ip_address='26.26.26.4', subnet_id='7b08899f-d866-418c-8080-b2e84efa4310'   | DOWN   |
	| 6d35f1bc-6e34-47e6-bf00-1146d6bef08e | vm1-port | fa:16:3e:b1:4e:cb | ip_address='26.26.26.12', subnet_id='7b08899f-d866-418c-8080-b2e84efa4310'  | ACTIVE |
	| 70a77356-5e56-4616-914a-bff2bee94d61 | vm2-port | fa:16:3e:55:13:74 | ip_address='26.26.26.7', subnet_id='7b08899f-d866-418c-8080-b2e84efa4310'   | ACTIVE |
	~~~

* We associated private IP to floating ips


	~~~
	(overcloud) [root@dell-r440-20 ~]# openstack floating ip list
	+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
	| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
	+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
	| 1d61e370-3745-4dd3-924a-40cf111ca232 | 10.10.10.112        | 26.26.26.12      | 6d35f1bc-6e34-47e6-bf00-1146d6bef08e | 377ea1cf-e9d8-47d5-b374-58e3c81f28aa | 2839e80d82f643afa981dea0db522b91 |
	| 48127265-8332-4fbe-bb8c-65c8a78db2bb | 10.10.10.108        | 26.26.26.7       | 70a77356-5e56-4616-914a-bff2bee94d61 | 377ea1cf-e9d8-47d5-b374-58e3c81f28aa | 2839e80d82f643afa981dea0db522b91 |
	| ab123cd9-17f1-49ea-a542-fa77a52a91c7 | 10.10.10.102        | 26.26.26.4       | 356f1ffd-69b1-456b-a48e-9beb35e6c98e | 377ea1cf-e9d8-47d5-b374-58e3c81f28aa | 2839e80d82f643afa981dea0db522b91 |
	+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+

* Create necessary security group rule and assign it to vm ports

* Create RHEL VMs using the ports.


	(overcloud) [root@dell-r440-20 ~]# openstack server list --long
	+--------------------------------------+------+--------+------------+-------------+-----------------------------------+------------+--------------------------------------+-------------+--------------------------------------+-------------------+-----------------------------------------+------------+
	| ID                                   | Name | Status | Task State | Power State | Networks                          | Image Name | Image ID                             | Flavor Name | Flavor ID                            | Availability Zone | Host                                    | Properties |
	+--------------------------------------+------+--------+------------+-------------+-----------------------------------+------------+--------------------------------------+-------------+--------------------------------------+-------------------+-----------------------------------------+------------+
	| ceaaab83-4d77-45e2-bf88-ddadfb08512d | vm2  | ACTIVE | None       | Running     | private=26.26.26.7, 10.10.10.108  | rhel       | 3f80e1ed-a0b2-430f-b9b4-7664281da465 | rhel        | 2107f813-5ab4-481e-bc5d-4500ee9e02d7 | nova              | overcloud-novacompute-dvr-1.localdomain |            |
	| 2d2ba1fe-ce5f-4bcd-bb62-f1f51e27b54f | vm1  | ACTIVE | None       | Running     | private=26.26.26.12, 10.10.10.112 | rhel       | 3f80e1ed-a0b2-430f-b9b4-7664281da465 | rhel        | 2107f813-5ab4-481e-bc5d-4500ee9e02d7 | nova              | overcloud-novacompute-dvr-0.localdomain |            |
	+--------------------------------------+------+--------+------------+-------------+-----------------------------------+------------+--------------------------------------+-------------+--------------------------------------+-------------------+-----------------------------------------+------------+



* VM1 currently have the vip:-

	~~~

	[root@vm1 ~]# ip a
	1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
	    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	    inet 127.0.0.1/8 scope host lo
	       valid_lft forever preferred_lft forever
	    inet6 ::1/128 scope host 
	       valid_lft forever preferred_lft forever
	2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast state UP group default qlen 1000
	    link/ether fa:16:3e:b1:4e:cb brd ff:ff:ff:ff:ff:ff
	    inet 26.26.26.12/24 brd 26.26.26.255 scope global noprefixroute dynamic eth0   ------> VM1 port ip
	       valid_lft 32403sec preferred_lft 32403sec
	    inet 26.26.26.4/32 scope global eth0    -----------------> VIP
	       valid_lft forever preferred_lft forever
	    inet6 fe80::f816:3eff:feb1:4ecb/64 scope link 
	       valid_lft forever preferred_lft forever

	conf:-

	[root@vm1 ~]# cat /etc/keepalived/keepalived.conf 
	vrrp_script chk_haproxy {
	  script "killall -0 haproxy" # check the haproxy process
	  interval 2 # every 2 seconds
	  weight 2 # add 2 points if OK
	}

	vrrp_instance VI_1 {
	  interface eth0 # interface to monitor
	  state MASTER # MASTER on haproxy, BACKUP on haproxy2
	  virtual_router_id 51
	  priority 101 # 101 on haproxy, 100 on haproxy2
	  virtual_ipaddress {
	    26.26.26.4 # virtual ip address
	  }
	  track_script {
	    chk_haproxy
	  }
	}
	~~~


* IP details and keepalive configuration of second vm:-
	~~~
	[root@vm2 ~]# ip a
	1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
	    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	    inet 127.0.0.1/8 scope host lo
	       valid_lft forever preferred_lft forever
	    inet6 ::1/128 scope host 
	       valid_lft forever preferred_lft forever
	2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast state UP group default qlen 1000
	    link/ether fa:16:3e:55:13:74 brd ff:ff:ff:ff:ff:ff
	    inet 26.26.26.7/24 brd 26.26.26.255 scope global noprefixroute dynamic eth0
	       valid_lft 31510sec preferred_lft 31510sec
	    inet6 fe80::f816:3eff:fe55:1374/64 scope link 
	       valid_lft forever preferred_lft forever
	[root@vm2 ~]# cat /etc/keepalived/keepalived.conf
	vrrp_script chk_haproxy {
	  script "killall -0 haproxy" # check the haproxy process
	  interval 2 # every 2 seconds
	  weight 2 # add 2 points if OK
	}

	vrrp_instance VI_1 {
	  interface eth0 # interface to monitor
	  state BACKUP # MASTER on haproxy, BACKUP on haproxy2
	  virtual_router_id 51
	  priority 100 # 101 on haproxy, 100 on haproxy2
	  virtual_ipaddress {
	    26.26.26.4 # virtual ip address
	  }
	  track_script {
	    chk_haproxy
	  }
	}
	~~~


* Interesting, VM1 and VIP private ip is reachable from ovnmetadata namespace from compute0 

	~~~
	[root@overcloud-novacompute-dvr-0 ~]# ip netns exec ovnmeta-b6611c71-9d30-424e-8dc5-a75fdcf6b30a ping 26.26.26.12
	PING 26.26.26.12 (26.26.26.12) 56(84) bytes of data.
	64 bytes from 26.26.26.12: icmp_seq=1 ttl=64 time=1.19 ms
	^C
	--- 26.26.26.12 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 1.199/1.199/1.199/0.000 ms



	[root@overcloud-novacompute-dvr-0 ~]# ip netns exec ovnmeta-b6611c71-9d30-424e-8dc5-a75fdcf6b30a ping 26.26.26.4   --------------> VIP IP reachable from ovnmeta namespace
	PING 26.26.26.4 (26.26.26.4) 56(84) bytes of data.
	64 bytes from 26.26.26.4: icmp_seq=1 ttl=64 time=0.991 ms
	64 bytes from 26.26.26.4: icmp_seq=2 ttl=64 time=0.913 ms
	^C
	--- 26.26.26.4 ping statistics ---
	2 packets transmitted, 2 received, 0% packet loss, time 1001ms
	rtt min/avg/max/mdev = 0.913/0.952/0.991/0.039 ms


* Also, VM2 IP is reachable from ovnmetadata namespace from compute1.

	[root@overcloud-novacompute-dvr-1 ~]# ip netns exec ovnmeta-b6611c71-9d30-424e-8dc5-a75fdcf6b30a ping 26.26.26.7
	PING 26.26.26.7 (26.26.26.7) 56(84) bytes of data.
	64 bytes from 26.26.26.7: icmp_seq=1 ttl=64 time=1.01 ms
	^C
	--- 26.26.26.7 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 1.016/1.016/1.016/0.000 ms

		~~~


* Floating ip associated with VM is pingable

	~~~
	(overcloud) [root@dell-r440-20 ~]# ping 10.10.10.108
	PING 10.10.10.108 (10.10.10.108) 56(84) bytes of data.
	64 bytes from 10.10.10.108: icmp_seq=1 ttl=63 time=1.64 ms
	^C
	--- 10.10.10.108 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 1.645/1.645/1.645/0.000 ms
	(overcloud) [root@dell-r440-20 ~]# ping 10.10.10.112
	PING 10.10.10.112 (10.10.10.112) 56(84) bytes of data.
	64 bytes from 10.10.10.112: icmp_seq=1 ttl=63 time=1.80 ms
	^C
	--- 10.10.10.112 ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 1.800/1.800/1.800/0.000 ms
	~~~

* But, Floating IP associated with vip is not pingable.

	~~~
	(overcloud) [root@dell-r440-20 ~]# ping 10.10.10.102
	PING 10.10.10.102 (10.10.10.102) 56(84) bytes of data.
	^C
	--- 10.10.10.102 ping statistics ---
	2 packets transmitted, 0 received, 100% packet loss, time 999ms
	~~~


It seems like some flow rules are missing for VIP.

	~~~
	[root@overcloud-controller-0 ~]#  ovn-nbctl --db=tcp:172.168.125.21:6641 lr-nat-list e41593b7-84e3-4cab-8daf-1d924b1d3e14
	TYPE             EXTERNAL_IP        LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
	dnat_and_snat    10.10.10.102       26.26.26.4                                                                       ----> VIP
	dnat_and_snat    10.10.10.108       26.26.26.7            fa:16:3e:70:e2:d7    70a77356-5e56-4616-914a-bff2bee94d61
	dnat_and_snat    10.10.10.112       26.26.26.12           fa:16:3e:f4:39:2f    6d35f1bc-6e34-47e6-bf00-1146d6bef08e
	snat             10.10.10.103       26.26.26.0/24
	~~~


Actual results:

Floating ip is not reachable


Expected results:

Floating IP should be reachable as It used to work ovs based environment.


Additional info:
Comment 20 Maciej Józefczyk 2019-06-21 08:21:11 UTC 
I verified that failover works properly with OSP13 + openvswitch-2.9.0-110.el7fdp.
Comment 24 errata-xmlrpc 2019-07-10 13:02:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1744