Bug 1503972 - [RFE] Make port pooling default
Summary: [RFE] Make port pooling default
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-kuryr-kubernetes
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Upstream M2
: 13.0 (Queens)
Assignee: Luis Tomas Bolivar
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On: 1503975
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-19 08:09 UTC by Antoni Segura Puimedon
Modified: 2018-06-27 13:38 UTC (History)
6 users (show)

Fixed In Version: openstack-kuryr-kubernetes-0.4.2-0.20180322192255.138c253.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:37:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 510157 None master: MERGED kuryr-kubernetes: Avoid neutron calls at recovering precreated ports (I635e7d3ac1a76b014dcfb1ef7539cfd56e73dddd) 2018-04-04 12:40:31 UTC
OpenStack gerrit 518502 None master: MERGED kuryr-kubernetes: Add readiness probe to kuryr-controller pod (Id47d3e7450551c19cb19d9278e459bd32bf364cf) 2018-04-04 12:40:24 UTC
OpenStack gerrit 529617 None master: MERGED kuryr-kubernetes: Updates ports pool doc information (If108b16cf998bd29fd51954e889982fb47ef4e8e) 2018-04-04 12:40:17 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:38:42 UTC

Description Antoni Segura Puimedon 2017-10-19 08:09:37 UTC
Description of problem:
Port pooling is a way to isolate the user from the latency inherent to Neutron bulk creation and deletion operations. The way it works is allocating and/or pre-allocating Neutron ports and assign them to the N worker nodes in the cluster. When a pod gets assigned to a cluster node two things can happen:

1. There's available ports assigned to the worker node where the Pod was assigned to run -> An existing vif (neutron port + binding info) will be assigned to the Pod and CNI will bind it.
2. There's no available ports. In this case, the Pool driver will delay the vif assignment and trigger a bulk creation in Neutron to re-stock the pool. 

While Port pooling exists in Kuryr-Kubernetes, it is optional at the moment and is in an experimental stage shape. In order to make it default we need:

a) Speed up controller loading of pre-allocated ports (currently takes ~14 minutes for 1500 ports).
b) Document a set of guidelines of how much pre-allocation should be done for different usage patterns,
c) Improve bulk operation flow
d) Add a readiness endpoint that gets set when all the ports of the pool are loaded.

Steps to Reproduce:
1. Configure OpenShift-Ansible to precreate 200 ports for the pool
2. Deploy kuryr with OpenShift-Ansible
3. oc -n kube-system get ConfigMap -o yaml

Actual results:
The ConfigMap for the Kuryr controller does not contain port pool configuration.

Expected results:
The ConfigMap contains port pool configuration and loads the pre-allocated ports.

The controller logs should show that the 200 ports were loaded.

Comment 8 Jon Uriarte 2018-05-08 17:19:22 UTC
Verified in version openstack-kuryr-kubernetes-controller-0.4.3-1.el7ost.noarch from puddle 20180502.1.

Verification steps (with openshift):

1. Leave kuryr configuration by default, but set openshift_kuryr_precreate_subports (in inventory/group_vars/all.yml):
  ## Ports pooling is enabled by default (with nested driver).
  ## To disable, uncomment the next:
  #kuryr_openstack_pool_driver: noop
  #
  ## You can also alter the port pooling defaults here
  #kuryr_openstack_pool_max: 0
  #kuryr_openstack_pool_min: 1
  #kuryr_openstack_pool_batch: 5
  #kuryr_openstack_pool_update_frequency: 20
  #
  ## You can pre-populate the pools with subports by uncommenting the next line
  ## and specifying the amount of subports you want to pre-create per
  ## origin-node, e.g.: openshift_kuryr_precreate_subports: 5
  openshift_kuryr_precreate_subports: 5
  #
  ## You can also change the default device owner for the precreated subports
  openshift_kuryr_device_owner: compute:kuryr

2. Deploy kuryr with OpenShift-Ansible
3. Check the ConfigMap contains port pool configuration
   $ oc -n openshift-infra get cm kuryr-config -o yaml
    ...
    [kubernetes]
    ...
    vif_pool_driver = nested
    ...

    [vif_pool]
              
    #
    # From kuryr_kubernetes
    #
    # Set a maximun amount of ports per pool. 0 to disable (integer value)
    ports_pool_max = 0                           
                                                                          
    # Set a target minimum size of the pool of ports (integer value)      
    ports_pool_min = 1                                                    
                                            
    # Number of ports to be created in a bulk request (integer value)     
    ports_pool_batch = 5                                                  
                                                  
    # Minimun interval (in seconds) between pool updates (integer value)  
    ports_pool_update_frequency = 20
    ...

4. Check precreated ports per trunk in openstack.

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk list
+--------------------------------------+------------------------------------+--------------------------------------+-------------+
| ID                                   | Name                               | Parent Port                          | Description |
+--------------------------------------+------------------------------------+--------------------------------------+-------------+
| 4c877f6f-1055-4149-a670-b38c4180bf6f | master-0.openshift.example.com     | a9829c12-dede-4e04-8b1b-05e515a750a1 |             |
| 531dba6e-2ef6-4434-97d6-12b5bfdb0329 | infra-node-0.openshift.example.com | 34269f29-4f35-4616-a2ff-a5e0f3a33984 |             |
| a70d7cdf-b607-43e0-bd16-3860083524ff | app-node-1.openshift.example.com   | a83cfad4-be92-418d-8798-2fe184515f6a |             |
| aa6bf2b4-bc93-4adb-b858-01d55a778b2a | app-node-0.openshift.example.com   | 93a7e85d-c410-4d34-ae04-7822223c74fc |             |
+--------------------------------------+------------------------------------+--------------------------------------+-------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show master-0.openshift.example.com
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2018-05-08T10:00:02Z                                                                             |
| description     |                                                                                                  |
| id              | 4c877f6f-1055-4149-a670-b38c4180bf6f                                                             |
| name            | master-0.openshift.example.com                                                                   |
| port_id         | a9829c12-dede-4e04-8b1b-05e515a750a1                                                             |
| project_id      | 6c07532860e641989bacc5583275080a                                                                 |
| revision_number | 5                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='026788d7-06db-4911-b4f8-6eee8cabeb06', segmentation_id='1', segmentation_type='vlan'    |
|                 | port_id='7a7ca31b-2178-45cd-bd3d-c581aa41eb10', segmentation_id='2', segmentation_type='vlan'    |
|                 | port_id='83cd5723-e2e3-4a59-8243-4e1aab704aa4', segmentation_id='3', segmentation_type='vlan'    |
|                 | port_id='660dd6b7-dfd9-491a-9db1-2115d6dbd4b3', segmentation_id='4', segmentation_type='vlan'    |
|                 | port_id='617846f3-c2d4-4b50-80ef-e6686119bb95', segmentation_id='5', segmentation_type='vlan'    |
|                 | port_id='8df19782-2539-403c-96cc-fc659abea238', segmentation_id='3444', segmentation_type='vlan' |
|                 | port_id='a0076398-5930-4ffb-ba51-335b6a5190dc', segmentation_id='3464', segmentation_type='vlan' |
|                 | port_id='4d2c4fbb-b078-4fae-ac31-76f04ba0839c', segmentation_id='3728', segmentation_type='vlan' |
|                 | port_id='e953854f-d1a0-430f-8cf9-ec81d37cf8b0', segmentation_id='3815', segmentation_type='vlan' |
|                 | port_id='e767d2d5-b79d-4ede-abab-f6bb9dafa149', segmentation_id='3948', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                                 |
| updated_at      | 2018-05-08T10:56:54Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show infra-node-0.openshift.example.com
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2018-05-08T09:59:35Z                                                                             |
| description     |                                                                                                  |
| id              | 531dba6e-2ef6-4434-97d6-12b5bfdb0329                                                             |
| name            | infra-node-0.openshift.example.com                                                               |
| port_id         | 34269f29-4f35-4616-a2ff-a5e0f3a33984                                                             |
| project_id      | 6c07532860e641989bacc5583275080a                                                                 |
| revision_number | 5                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='66aa7ce2-83c2-4b07-8fa5-bb81270cf445', segmentation_id='1', segmentation_type='vlan'    |
|                 | port_id='8ea29a23-7895-4e40-8441-9d521821de90', segmentation_id='2', segmentation_type='vlan'    |
|                 | port_id='8326327a-5e08-45c8-b448-f7e8c61feb0b', segmentation_id='3', segmentation_type='vlan'    |
|                 | port_id='a66883d5-210a-49c9-8924-58ed9512f976', segmentation_id='4', segmentation_type='vlan'    |
|                 | port_id='0a0d36f8-a55a-44dd-96a0-e93037caf6c8', segmentation_id='5', segmentation_type='vlan'    |
|                 | port_id='07f13bae-d44d-44d6-8b0a-47838a266bbd', segmentation_id='1231', segmentation_type='vlan' |
|                 | port_id='d4f377cf-1bb3-40c4-b1c6-968509489a30', segmentation_id='1818', segmentation_type='vlan' |
|                 | port_id='86d280f3-bf1d-4c73-991a-64086bd7a063', segmentation_id='1851', segmentation_type='vlan' |
|                 | port_id='7288719a-fdd4-43db-9f1a-df3cf712cf08', segmentation_id='1927', segmentation_type='vlan' |
|                 | port_id='dd16a816-fc6c-4a47-833f-9fe79ead806e', segmentation_id='3814', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                                 |
| updated_at      | 2018-05-08T10:56:57Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show app-node-0.openshift.example.com
+-----------------+-----------------------------------------------------------------------------------------------+
| Field           | Value                                                                                         |
+-----------------+-----------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                            |
| created_at      | 2018-05-08T09:59:34Z                                                                          |
| description     |                                                                                               |
| id              | aa6bf2b4-bc93-4adb-b858-01d55a778b2a                                                          |
| name            | app-node-0.openshift.example.com                                                              |
| port_id         | 93a7e85d-c410-4d34-ae04-7822223c74fc                                                          |
| project_id      | 6c07532860e641989bacc5583275080a                                                              |
| revision_number | 2                                                                                             |
| status          | ACTIVE                                                                                        |
| sub_ports       | port_id='1d76be7c-b711-4fe2-a941-1922318e3475', segmentation_id='1', segmentation_type='vlan' |
|                 | port_id='89caadf0-9ed4-4959-8638-4f90e3d90d6d', segmentation_id='2', segmentation_type='vlan' |
|                 | port_id='ada9a8ad-fbf3-4aac-81d2-b27a9e9910a6', segmentation_id='3', segmentation_type='vlan' |
|                 | port_id='8cabf3bf-d403-4a43-b3c6-7bd6833b7e6b', segmentation_id='4', segmentation_type='vlan' |
|                 | port_id='716bde48-2ec2-4aab-8ee4-d3e3c46ca372', segmentation_id='5', segmentation_type='vlan' |
| tags            | []                                                                                            |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                              |
| updated_at      | 2018-05-08T09:59:46Z                                                                          |
+-----------------+-----------------------------------------------------------------------------------------------+

(overcloud) [cloud-user@ansible-host ~]$ openstack network trunk show app-node-1.openshift.example.com
+-----------------+-----------------------------------------------------------------------------------------------+
| Field           | Value                                                                                         |
+-----------------+-----------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                            |
| created_at      | 2018-05-08T09:59:34Z                                                                          |
| description     |                                                                                               |
| id              | a70d7cdf-b607-43e0-bd16-3860083524ff                                                          |
| name            | app-node-1.openshift.example.com                                                              |
| port_id         | a83cfad4-be92-418d-8798-2fe184515f6a                                                          |
| project_id      | 6c07532860e641989bacc5583275080a                                                              |
| revision_number | 2                                                                                             |
| status          | ACTIVE                                                                                        |
| sub_ports       | port_id='ec61324d-e6e5-4abf-8f3c-ab399445b05c', segmentation_id='1', segmentation_type='vlan' |
|                 | port_id='e1da30ac-0a6b-4efc-8346-32e59eb79ca2', segmentation_id='2', segmentation_type='vlan' |
|                 | port_id='546cc02a-a33d-4a24-91eb-834f4cc03e83', segmentation_id='3', segmentation_type='vlan' |
|                 | port_id='7ea4a036-33c8-4966-b0b9-e11993a92bfb', segmentation_id='4', segmentation_type='vlan' |
|                 | port_id='46ec2829-5d1e-417f-adf2-9e071b6f7801', segmentation_id='5', segmentation_type='vlan' |
| tags            | []                                                                                            |
| tenant_id       | 6c07532860e641989bacc5583275080a                                                              |
| updated_at      | 2018-05-08T09:59:52Z                                                                          |
+-----------------+-----------------------------------------------------------------------------------------------+

5 subports with vlan from 1 to 5 have been precreated in each openshift node.

Comment 10 errata-xmlrpc 2018-06-27 13:37:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.