Bug 1503972

Summary:	[RFE] Make port pooling default
Product:	Red Hat OpenStack	Reporter:	Antoni Segura Puimedon <asegurap>
Component:	openstack-kuryr-kubernetes	Assignee:	Luis Tomas Bolivar <ltomasbo>
Status:	CLOSED ERRATA	QA Contact:	Jon Uriarte <juriarte>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	13.0 (Queens)	CC:	asegurap, jschluet, juriarte, lpeer, sgordon, tsedovic
Target Milestone:	Upstream M2	Keywords:	FutureFeature, Triaged
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-kuryr-kubernetes-0.4.2-0.20180322192255.138c253.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-27 13:37:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1503975
Bug Blocks:

Description Antoni Segura Puimedon 2017-10-19 08:09:37 UTC

Description of problem:
Port pooling is a way to isolate the user from the latency inherent to Neutron bulk creation and deletion operations. The way it works is allocating and/or pre-allocating Neutron ports and assign them to the N worker nodes in the cluster. When a pod gets assigned to a cluster node two things can happen:

1. There's available ports assigned to the worker node where the Pod was assigned to run -> An existing vif (neutron port + binding info) will be assigned to the Pod and CNI will bind it.
2. There's no available ports. In this case, the Pool driver will delay the vif assignment and trigger a bulk creation in Neutron to re-stock the pool.

While Port pooling exists in Kuryr-Kubernetes, it is optional at the moment and is in an experimental stage shape. In order to make it default we need:

a) Speed up controller loading of pre-allocated ports (currently takes ~14 minutes for 1500 ports).
b) Document a set of guidelines of how much pre-allocation should be done for different usage patterns,
c) Improve bulk operation flow
d) Add a readiness endpoint that gets set when all the ports of the pool are loaded.

Steps to Reproduce:
1. Configure OpenShift-Ansible to precreate 200 ports for the pool
2. Deploy kuryr with OpenShift-Ansible
3. oc -n kube-system get ConfigMap -o yaml

Actual results:
The ConfigMap for the Kuryr controller does not contain port pool configuration.

Expected results:
The ConfigMap contains port pool configuration and loads the pre-allocated ports.

The controller logs should show that the 200 ports were loaded.

Comment 8 Jon Uriarte 2018-05-08 17:19:22 UTC

Verified in version openstack-kuryr-kubernetes-controller-0.4.3-1.el7ost.noarch from puddle 20180502.1.

Verification steps (with openshift):

1. Leave kuryr configuration by default, but set openshift_kuryr_precreate_subports (in inventory/group_vars/all.yml):
  ## Ports pooling is enabled by default (with nested driver).
  ## To disable, uncomment the next:
  #kuryr_openstack_pool_driver: noop
  #
  ## You can also alter the port pooling defaults here
  #kuryr_openstack_pool_max: 0
  #kuryr_openstack_pool_min: 1
  #kuryr_openstack_pool_batch: 5
  #kuryr_openstack_pool_update_frequency: 20
  #
  ## You can pre-populate the pools with subports by uncommenting the next line
  ## and specifying the amount of subports you want to pre-create per
  ## origin-node, e.g.: openshift_kuryr_precreate_subports: 5
  openshift_kuryr_precreate_subports: 5
  #
  ## You can also change the default device owner for the precreated subports
  openshift_kuryr_device_owner: compute:kuryr

2. Deploy kuryr with OpenShift-Ansible
3. Check the ConfigMap contains port pool configuration
   $ oc -n openshift-infra get cm kuryr-config -o yaml
    ...
    [kubernetes]
    ...
    vif_pool_driver = nested
    ...

    [vif_pool]
              
    #
    # From kuryr_kubernetes
    #
    # Set a maximun amount of ports per pool. 0 to disable (integer value)
    ports_pool_max = 0                           
                                                                          
    # Set a target minimum size of the pool of ports (integer value)      
    ports_pool_min = 1                                                    
                                            
    # Number of ports to be created in a bulk request (integer value)     
    ports_pool_batch = 5                                                  
                                                  
    # Minimun interval (in seconds) between pool updates (integer value)  
    ports_pool_update_frequency = 20
    ...

4. Check precreated ports per trunk in openstack.

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk list
+--------------------------------------+------------------------------------+--------------------------------------+-------------+
| ID                                   | Name                               | Parent Port                          | Description |
+--------------------------------------+------------------------------------+--------------------------------------+-------------+
| 4c877f6f-1055-4149-a670-b38c4180bf6f | master-0.openshift.example.com     | a9829c12-dede-4e04-8b1b-05e515a750a1 |             |
| 531dba6e-2ef6-4434-97d6-12b5bfdb0329 | infra-node-0.openshift.example.com | 34269f29-4f35-4616-a2ff-a5e0f3a33984 |             |
| a70d7cdf-b607-43e0-bd16-3860083524ff | app-node-1.openshift.example.com   | a83cfad4-be92-418d-8798-2fe184515f6a |             |
| aa6bf2b4-bc93-4adb-b858-01d55a778b2a | app-node-0.openshift.example.com   | 93a7e85d-c410-4d34-ae04-7822223c74fc |             |
+--------------------------------------+------------------------------------+--------------------------------------+-------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show master-0.openshift.example.com
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2018-05-08T10:00:02Z                                                                             |
| description     |                                                                                                  |
| id              | 4c877f6f-1055-4149-a670-b38c4180bf6f                                                             |
| name            | master-0.openshift.example.com                                                                   |
| port_id         | a9829c12-dede-4e04-8b1b-05e515a750a1                                                             |
| project_id      | 6c07532860e641989bacc5583275080a                                                                 |
| revision_number | 5                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='026788d7-06db-4911-b4f8-6eee8cabeb06', segmentation_id='1', segmentation_type='vlan'    |
|                 | port_id='7a7ca31b-2178-45cd-bd3d-c581aa41eb10', segmentation_id='2', segmentation_type='vlan'    |
|                 | port_id='83cd5723-e2e3-4a59-8243-4e1aab704aa4', segmentation_id='3', segmentation_type='vlan'    |
|                 | port_id='660dd6b7-dfd9-491a-9db1-2115d6dbd4b3', segmentation_id='4', segmentation_type='vlan'    |
|                 | port_id='617846f3-c2d4-4b50-80ef-e6686119bb95', segmentation_id='5', segmentation_type='vlan'    |
|                 | port_id='8df19782-2539-403c-96cc-fc659abea238', segmentation_id='3444', segmentation_type='vlan' |
|                 | port_id='a0076398-5930-4ffb-ba51-335b6a5190dc', segmentation_id='3464', segmentation_type='vlan' |
|                 | port_id='4d2c4fbb-b078-4fae-ac31-76f04ba0839c', segmentation_id='3728', segmentation_type='vlan' |
|                 | port_id='e953854f-d1a0-430f-8cf9-ec81d37cf8b0', segmentation_id='3815', segmentation_type='vlan' |
|                 | port_id='e767d2d5-b79d-4ede-abab-f6bb9dafa149', segmentation_id='3948', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                                 |
| updated_at      | 2018-05-08T10:56:54Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show infra-node-0.openshift.example.com
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2018-05-08T09:59:35Z                                                                             |
| description     |                                                                                                  |
| id              | 531dba6e-2ef6-4434-97d6-12b5bfdb0329                                                             |
| name            | infra-node-0.openshift.example.com                                                               |
| port_id         | 34269f29-4f35-4616-a2ff-a5e0f3a33984                                                             |
| project_id      | 6c07532860e641989bacc5583275080a                                                                 |
| revision_number | 5                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='66aa7ce2-83c2-4b07-8fa5-bb81270cf445', segmentation_id='1', segmentation_type='vlan'    |
|                 | port_id='8ea29a23-7895-4e40-8441-9d521821de90', segmentation_id='2', segmentation_type='vlan'    |
|                 | port_id='8326327a-5e08-45c8-b448-f7e8c61feb0b', segmentation_id='3', segmentation_type='vlan'    |
|                 | port_id='a66883d5-210a-49c9-8924-58ed9512f976', segmentation_id='4', segmentation_type='vlan'    |
|                 | port_id='0a0d36f8-a55a-44dd-96a0-e93037caf6c8', segmentation_id='5', segmentation_type='vlan'    |
|                 | port_id='07f13bae-d44d-44d6-8b0a-47838a266bbd', segmentation_id='1231', segmentation_type='vlan' |
|                 | port_id='d4f377cf-1bb3-40c4-b1c6-968509489a30', segmentation_id='1818', segmentation_type='vlan' |
|                 | port_id='86d280f3-bf1d-4c73-991a-64086bd7a063', segmentation_id='1851', segmentation_type='vlan' |
|                 | port_id='7288719a-fdd4-43db-9f1a-df3cf712cf08', segmentation_id='1927', segmentation_type='vlan' |
|                 | port_id='dd16a816-fc6c-4a47-833f-9fe79ead806e', segmentation_id='3814', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                                 |
| updated_at      | 2018-05-08T10:56:57Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show app-node-0.openshift.example.com
+-----------------+-----------------------------------------------------------------------------------------------+
| Field           | Value                                                                                         |
+-----------------+-----------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                            |
| created_at      | 2018-05-08T09:59:34Z                                                                          |
| description     |                                                                                               |
| id              | aa6bf2b4-bc93-4adb-b858-01d55a778b2a                                                          |
| name            | app-node-0.openshift.example.com                                                              |
| port_id         | 93a7e85d-c410-4d34-ae04-7822223c74fc                                                          |
| project_id      | 6c07532860e641989bacc5583275080a                                                              |
| revision_number | 2                                                                                             |
| status          | ACTIVE                                                                                        |
| sub_ports       | port_id='1d76be7c-b711-4fe2-a941-1922318e3475', segmentation_id='1', segmentation_type='vlan' |
|                 | port_id='89caadf0-9ed4-4959-8638-4f90e3d90d6d', segmentation_id='2', segmentation_type='vlan' |
|                 | port_id='ada9a8ad-fbf3-4aac-81d2-b27a9e9910a6', segmentation_id='3', segmentation_type='vlan' |
|                 | port_id='8cabf3bf-d403-4a43-b3c6-7bd6833b7e6b', segmentation_id='4', segmentation_type='vlan' |
|                 | port_id='716bde48-2ec2-4aab-8ee4-d3e3c46ca372', segmentation_id='5', segmentation_type='vlan' |
| tags            | []                                                                                            |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                              |
| updated_at      | 2018-05-08T09:59:46Z                                                                          |
+-----------------+-----------------------------------------------------------------------------------------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show app-node-1.openshift.example.com
+-----------------+-----------------------------------------------------------------------------------------------+
| Field           | Value                                                                                         |
+-----------------+-----------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                            |
| created_at      | 2018-05-08T09:59:34Z                                                                          |
| description     |                                                                                               |
| id              | a70d7cdf-b607-43e0-bd16-3860083524ff                                                          |
| name            | app-node-1.openshift.example.com                                                              |
| port_id         | a83cfad4-be92-418d-8798-2fe184515f6a                                                          |
| project_id      | 6c07532860e641989bacc5583275080a                                                              |
| revision_number | 2                                                                                             |
| status          | ACTIVE                                                                                        |
| sub_ports       | port_id='ec61324d-e6e5-4abf-8f3c-ab399445b05c', segmentation_id='1', segmentation_type='vlan' |
|                 | port_id='e1da30ac-0a6b-4efc-8346-32e59eb79ca2', segmentation_id='2', segmentation_type='vlan' |
|                 | port_id='546cc02a-a33d-4a24-91eb-834f4cc03e83', segmentation_id='3', segmentation_type='vlan' |
|                 | port_id='7ea4a036-33c8-4966-b0b9-e11993a92bfb', segmentation_id='4', segmentation_type='vlan' |
|                 | port_id='46ec2829-5d1e-417f-adf2-9e071b6f7801', segmentation_id='5', segmentation_type='vlan' |
| tags            | []                                                                                            |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                              |
| updated_at      | 2018-05-08T09:59:52Z                                                                          |
+-----------------+-----------------------------------------------------------------------------------------------+

5 subports with vlan from 1 to 5 have been precreated in each openshift node.

Comment 10 errata-xmlrpc 2018-06-27 13:37:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086