Bug 1743302

Summary:	OpenStack, Kuryr: Fix conflicts with Octavia VRRP ports
Product:	OpenShift Container Platform	Reporter:	Michał Dulko <mdulko>
Component:	Networking	Assignee:	Michał Dulko <mdulko>
Networking sub component:	kuryr	QA Contact:	Jon Uriarte <juriarte>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	aos-bugs, gcheresh, juriarte, ltomasbo, piqin, racedoro, xtian
Version:	4.2.0	Keywords:	Triaged
Target Milestone:	---
Target Release:	4.2.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-16 06:36:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Michał Dulko 2019-08-19 15:17:41 UTC

Description of problem:
When running with Kuryr, OpenStack Octavia is used to create loadbalancers for OpenShift Services. With the default Amphora driver it uses two IPs from the service subnet for each loadbalancer, the second being used as VRRP port. The issue with that is that OpenShift uses it's own IPAM and doesn't know about Octavia's habits leading to conflicts if IP chosen by OpenShift is already taken by one of Amphora VRRP ports.

Version-Release number of selected component (if applicable):
4.2

How reproducible:
Default service network is /16, so it's not happening that often, but obviously bug is there.

Steps to Reproduce:
N/A, bug would manifest randomly on any Kuryr-based installation. You can use a smaller service network to increase probability of it happening.

Actual results:
It may happen that a newly created service will get an IP already allocated in service subnet by one of Amphora VRRP ports.

Expected results:
No IP conflicts.

Additional info:
This was agreed to require changes to openshift/api to add an option to KuryrConfig structure by the cluster-network-operator devs.

Comment 1 Michał Dulko 2019-08-21 09:53:48 UTC

Back to ASSIGNED, it requires https://github.com/openshift/cluster-network-operator/pull/275 to be fully fixed.

Comment 4 Jon Uriarte 2019-10-04 10:58:55 UTC

Verified on 4.2.0-0.nightly-2019-10-02-150642 on top of OSP 13 2019-10-01.1 puddle.

With install-config.yaml:
networking:
  clusterNetworks:
  - cidr:             10.128.0.0/14
    hostSubnetLength: 9
  serviceCIDR: 172.30.0.0/16
  machineCIDR: 10.196.0.0/16
  type: "Kuryr"

172.30.0.0/16 network is used now for services VIPs, and 172.31.0.0-172.31.255.253
for Octavia VRRP IPs, so they cannot collide.

$ oc get networks.config.openshift.io cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: "2019-10-03T11:52:38Z"
  generation: 2
  name: cluster
  resourceVersion: "2340"
  selfLink: /apis/config.openshift.io/v1/networks/cluster
  uid: 4c427b8f-e5d4-11e9-b2e7-fa163e91b538
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: Kuryr
  serviceNetwork:
  - 172.30.0.0/16
status:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: Kuryr
  serviceNetwork:
  - 172.30.0.0/16


$ oc get svc -A
NAMESPACE                                               NAME                               TYPE           CLUSTER-IP    
default                                                 kubernetes                         ClusterIP      172.30.0.1    
default                                                 kuryr-service-1211953743           LoadBalancer   172.30.208.181
default                                                 kuryr-service-1641305923           LoadBalancer   172.30.150.87 
kube-system                                             kubelet                            ClusterIP      None          
openshift-apiserver-operator                            metrics                            ClusterIP      172.30.243.171
openshift-apiserver                                     api                                ClusterIP      172.30.177.241
openshift-authentication-operator                       metrics                            ClusterIP      172.30.177.91 
openshift-authentication                                oauth-openshift                    ClusterIP      172.30.91.243 
openshift-cloud-credential-operator                     controller-manager-service         ClusterIP      172.30.238.255
openshift-cluster-version                               cluster-version-operator           ClusterIP      172.30.143.244
openshift-console-operator                              metrics                            ClusterIP      172.30.165.184
openshift-console                                       console                            ClusterIP      172.30.197.10 
openshift-console                                       downloads                          ClusterIP      172.30.120.142
openshift-controller-manager-operator                   metrics                            ClusterIP      172.30.63.85  
openshift-controller-manager                            controller-manager                 ClusterIP      172.30.117.244
openshift-dns                                           dns-default                        ClusterIP      172.30.0.10   
openshift-etcd                                          etcd                               ClusterIP      172.30.164.7  
openshift-etcd                                          host-etcd                          ClusterIP      None          
openshift-image-registry                                image-registry                     ClusterIP      172.30.45.123 
openshift-ingress                                       router-internal-default            ClusterIP      172.30.246.35 
openshift-kube-apiserver-operator                       metrics                            ClusterIP      172.30.69.107 
openshift-kube-apiserver                                apiserver                          ClusterIP      172.30.155.242
openshift-kube-controller-manager-operator              metrics                            ClusterIP      172.30.19.96  
openshift-kube-controller-manager                       kube-controller-manager            ClusterIP      172.30.27.205 
openshift-kube-scheduler-operator                       metrics                            ClusterIP      172.30.255.101
openshift-kube-scheduler                                scheduler                          ClusterIP      172.30.228.254
openshift-kuryr                                         kuryr-dns-admission-controller     ClusterIP      172.30.39.153 
openshift-machine-api                                   cluster-autoscaler-operator        ClusterIP      172.30.145.195
openshift-machine-api                                   machine-api-operator               ClusterIP      172.30.128.200
openshift-marketplace                                   marketplace-operator-metrics       ClusterIP      172.30.207.2  
openshift-monitoring                                    alertmanager-main                  ClusterIP      172.30.89.29  
openshift-monitoring                                    alertmanager-operated              ClusterIP      None          
openshift-monitoring                                    cluster-monitoring-operator        ClusterIP      None          
openshift-monitoring                                    grafana                            ClusterIP      172.30.148.179
openshift-monitoring                                    kube-state-metrics                 ClusterIP      None          
openshift-monitoring                                    node-exporter                      ClusterIP      None          
openshift-monitoring                                    openshift-state-metrics            ClusterIP      None          
openshift-monitoring                                    prometheus-adapter                 ClusterIP      172.30.101.230
openshift-monitoring                                    prometheus-k8s                     ClusterIP      172.30.148.47 
openshift-monitoring                                    prometheus-operated                ClusterIP      None          
openshift-monitoring                                    prometheus-operator                ClusterIP      None          
openshift-monitoring                                    telemeter-client                   ClusterIP      None          
openshift-multus                                        multus-admission-controller        ClusterIP      172.30.131.71 
openshift-operator-lifecycle-manager                    catalog-operator-metrics           ClusterIP      172.30.195.117
openshift-operator-lifecycle-manager                    olm-operator-metrics               ClusterIP      172.30.3.104  
openshift-operator-lifecycle-manager                    v1-packages-operators-coreos-com   ClusterIP      172.30.128.187
openshift-service-catalog-apiserver-operator            metrics                            ClusterIP      172.30.167.87 
openshift-service-catalog-controller-manager-operator   metrics                            ClusterIP      172.30.174.251
test                                                    pod1                               ClusterIP      172.30.129.4  

$ openstack subnet list | grep service
...
| 362483d6-c761-4db1-be11-bbcb6e7b025a | ostest-mp284-kuryr-service-subnet  | a52bd384-c1af-46ea-bea4-df41d9202e34 | 172.30.0.0/15   |


$ openstack subnet show ostest-mp284-kuryr-service-subnet
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 172.31.0.0-172.31.255.253            |
| cidr              | 172.30.0.0/15                        |
| created_at        | 2019-10-03T11:53:17Z                 |
| description       |                                      |
| dns_nameservers   |                                      |
| enable_dhcp       | False                                |
| gateway_ip        | 172.31.255.254                       |
| host_routes       |                                      |
| id                | 362483d6-c761-4db1-be11-bbcb6e7b025a |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | ostest-mp284-kuryr-service-subnet    |
| network_id        | a52bd384-c1af-46ea-bea4-df41d9202e34 |
| prefix_length     | None                                 |
| project_id        | 4d589eb96cb04a4598056bc3679b63dc     |
| revision_number   | 1                                    |
| segment_id        | None                                 |
| service_types     |                                      |
| subnetpool_id     | None                                 |
| tags              | openshiftClusterID=ostest-mp284      |
| updated_at        | 2019-10-03T11:53:17Z                 |
+-------------------+--------------------------------------+

$ openstack port list | grep vrrp
| octavia-lb-vrrp-035294a2-4808-4766-8369-cd1093d286c5 | ip_address='172.31.0.27', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-25d4d45c-87fe-4a7a-9017-97dff8173d8b | ip_address='172.31.0.17', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-5fcd6a98-c1ae-4431-af58-87179d2dd740 | ip_address='172.31.0.18', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-20bd5855-fac8-4152-aa27-1838c4f0fe69 | ip_address='172.31.0.12', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-505f7fe7-bad8-496e-87fb-92f8412b55d6 | ip_address='172.31.0.3', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-445719c9-283d-4450-bf8c-df3770a50fed | ip_address='172.31.0.11', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-d9dcf935-96bd-4157-ba40-c659cfff6b09 | ip_address='172.31.0.48', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-b10d1c3d-768d-4556-9faf-8e6d15285c16 | ip_address='172.31.0.23', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-a8ab2c51-c6d4-45a4-a116-db7d46d67f1b | ip_address='172.31.0.31', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-6d7f9ca0-4273-4949-b56c-6f82cd3fa8ed | ip_address='172.31.0.8', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-f98e8261-735d-454f-9205-52ce76d74b63 | ip_address='172.31.0.30', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-4c969cc2-a02e-463b-aa6b-23c0d3586402 | ip_address='172.31.0.0', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-73dc78ae-f426-42f4-b2ec-a04e89bf8859 | ip_address='172.31.0.16', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-6b213f38-2d1e-4e02-9e42-6842ee5ed53b | ip_address='172.31.0.42', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-0ba60e5a-0ea7-4bd0-b9a0-7153ed71867c | ip_address='172.31.0.25', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-1af764bb-3ed1-42a5-ad9a-193357bce49c | ip_address='172.31.0.20', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-dd5ad427-bfd4-4c6f-85cc-ea8531d9ad4a | ip_address='172.31.0.43', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-b71bc775-d83f-4ed7-af75-c13618776cbb | ip_address='172.31.0.33', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-e3a13521-0903-48d4-9e00-23e6c8793289 | ip_address='172.31.0.9', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-c538f195-1b37-4c99-b41a-ef6b69196854 | ip_address='172.31.0.21', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-f3f40e75-457b-4e17-998c-6d00884b21b6 | ip_address='172.31.0.56', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-2404ab71-3c09-4268-8905-ba85c10c245d | ip_address='172.31.0.1', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-5b69ea1b-b0b3-4464-8bf3-f70cd4f25aea | ip_address='172.31.0.4', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-5d2c5ff7-e897-47cd-a765-4d09c0390d1d | ip_address='172.31.0.47', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-0b360981-7c8c-46fe-9ba5-c9f91de3c7b6 | ip_address='172.31.0.10', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-85ca22e8-40b9-4e14-ace1-8aa941831b7d | ip_address='172.31.0.26', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-77824b8f-dc86-4518-98cd-de14553deca4 | ip_address='172.31.0.32', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-61510efd-a794-4bcc-8950-b769e885b990 | ip_address='172.31.0.19', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-b427c840-0a2a-4342-87d7-a1be2ecaff55 | ip_address='172.31.0.35', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-aced3ca1-0bbd-40de-83a6-9871d5cc7979 | ip_address='172.31.0.5', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-3edade9f-2f82-4419-af40-ba42df23572d | ip_address='172.31.0.24', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-ae0a31c8-2b25-433f-8fd8-33e9d72d1196 | ip_address='172.31.0.7', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-d0ee0a73-190a-45fe-bc42-7f476a7f199c | ip_address='172.31.0.6', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-c0b880c5-9ead-466d-94cc-41edcd2bd68b | ip_address='172.31.0.29', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-a6e6dcb9-c329-43c9-ab7d-76009aa324ea | ip_address='172.31.0.13', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-27bec36e-8ca6-49eb-862b-6de2dc43e0b1 | ip_address='172.31.0.62', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-dd0146a6-6c69-4fdb-9081-794fa36a3f6a | ip_address='172.31.0.49', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-50f87283-860b-4ca0-8af0-9bf2f4a77e26 | ip_address='172.31.0.41', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-9963a48c-85b6-4157-a9de-b1d45350e44a | ip_address='172.31.0.2', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a'  | ACTIVE |
| octavia-lb-vrrp-7eefe6f3-baea-46fd-9e6b-652fce422798 | ip_address='172.31.0.22', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |
| octavia-lb-vrrp-77b93190-4d19-4617-a30a-511158f15ec8 | ip_address='172.31.0.14', subnet_id='362483d6-c761-4db1-be11-bbcb6e7b025a' | ACTIVE |

Comment 5 errata-xmlrpc 2019-10-16 06:36:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922