Bug 1703947 - Using remote_group_id affects the subports attachment to the trunks
Summary: Using remote_group_id affects the subports attachment to the trunks
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.11.z
Assignee: Luis Tomas Bolivar
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-29 07:53 UTC by Luis Tomas Bolivar
Modified: 2019-06-26 09:08 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-26 09:08:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift openshift-ansible pull 11560 None closed Replace remote_group_id by remote_ip_prefix 2020-06-23 22:46:31 UTC
Red Hat Product Errata RHBA-2019:1605 None None None 2019-06-26 09:08:23 UTC

Description Luis Tomas Bolivar 2019-04-29 07:53:45 UTC
In case when on one compute node there is many ports which are using same security group and this security group uses rule to allow traffic from ports with given SG adding new port to host is very slow, as all the related ports needs to be taken into account.

We tested it on compute with 170 trunk supports and it took about 5 minutes for new port to become ACTIVE.

In order to avoid this problem until [1] is solved, there is a need for moving away from remote_group_id when possible, in favor of remote_prefix_ip

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1703467

Comment 1 Jon Uriarte 2019-05-17 18:06:10 UTC
Verified in openshift-ansible-3.11.115-1 on top of OSP13 2019-05-15.1 puddle.

Verification steps:
- Deploy OSP 13 with Octavia and ML2/OVS Neutron backend
- Deploy OCP on top with Kuryr SDN, and without namespace isolation
- Create 150 pods (in 3 different projects) so many ports are created

[openshift@master-0 ~]$
oc new-project test
oc run --image kuryr/demo demo
oc scale dc/demo --replicas=50                                                                                                                                                             

oc new-project test2
oc run --image kuryr/demo demo                                                                                                                                                             
oc scale dc/demo --replicas=50                                                                                                                                                             

oc new-project test3
oc run --image kuryr/demo demo                                                                                                                                                             
oc scale dc/demo --replicas=50                                                                    

oc get pods --all-namespaces | grep test | grep Running | wc -l
150

- Check the SG for pod/service is using remote_ip_prefix
(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack security group list
+--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+
| ID                                   | Name                                                       | Description                                                                                                                                           | Project                          |
+--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+
| 45468573-3b83-44a8-af7d-ab1f35a65fd6 | openshift-ansible-openshift.example.com-infra-secgrp       | Security group for openshift.example.com OpenShift infrastructure cluster nodes                                                                       | bad742dd55e64ee1a2f7fdd2f0818bad |
| 4610f3f8-bbe9-4d1c-99eb-dc2d7982b80a | openshift-ansible-openshift.example.com-pod-service-secgrp | Give services and nodes access to the pods                                                                                                            | bad742dd55e64ee1a2f7fdd2f0818bad |
| 56173aac-d734-4735-9d5d-ff68dfe8894e | default                                                    | Default security group                                                                                                                                | bad742dd55e64ee1a2f7fdd2f0818bad |
| 5746452e-a079-4254-90bb-f02bacad8d43 | openshift-ansible-openshift.example.com-node-secgrp        | Security group for openshift.example.com OpenShift cluster nodes                                                                                      | bad742dd55e64ee1a2f7fdd2f0818bad |
| 734b36b0-1e54-4dda-ab3c-1f7765db0aac | openshift-ansible-openshift.example.com-lb-secgrp          | Security group for openshift.example.com cluster Load Balancer                                                                                        | bad742dd55e64ee1a2f7fdd2f0818bad |
| a1267a46-826e-40de-8261-a3ac16b9b4ae | openshift-ansible-openshift.example.com-etcd-secgrp        | Security group for openshift.example.com etcd cluster                                                                                                 | bad742dd55e64ee1a2f7fdd2f0818bad |
| a510e6ee-085c-4aeb-9a21-db75f1692bae | openshift-ansible-openshift.example.com-common-secgrp      | Basic ssh/icmp security group for openshift.example.com OpenShift cluster                                                                             | bad742dd55e64ee1a2f7fdd2f0818bad |
| aeab021c-14ff-4a27-aac8-54e741a2aaac | openshift-ansible-openshift.example.com-master-secgrp      | Security group for openshift.example.com OpenShift cluster master                                                                                     | bad742dd55e64ee1a2f7fdd2f0818bad |
| dcd9cafc-3d6b-4aee-b524-8ce5a18e263a | secgroup_openshift_dns                                     | ir: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-osasinfra-shiftstack_on_vms-13-customized-job-hybrid/82/ at  2019-05-17T14:58:05Z | bad742dd55e64ee1a2f7fdd2f0818bad |
| feacb4d7-9b38-4df0-870c-806fef55990f | secgroup_openshift                                         | ir: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-osasinfra-shiftstack_on_vms-13-customized-job-hybrid/82/ at  2019-05-17T14:58:05Z | bad742dd55e64ee1a2f7fdd2f0818bad |
+--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+


(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack security group show openshift-ansible-openshift.example.com-pod-service-secgrp
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                      |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at      | 2019-05-17T15:56:30Z                                                                                                                                                                       |
| description     | Give services and nodes access to the pods                                                                                                                                                 |
| id              | 4610f3f8-bbe9-4d1c-99eb-dc2d7982b80a                                                                                                                                                       |
| name            | openshift-ansible-openshift.example.com-pod-service-secgrp                                                                                                                                 |
| project_id      | bad742dd55e64ee1a2f7fdd2f0818bad                                                                                                                                                           |
| revision_number | 5                                                                                                                                                                                          |
| rules           | created_at='2019-05-17T15:56:31Z', direction='ingress', ethertype='IPv4', id='2e0324a4-f1bc-4ed4-b744-7ad7787147f8', remote_ip_prefix='10.11.0.0/16', updated_at='2019-05-17T15:56:31Z'    |
|                 | created_at='2019-05-17T15:56:31Z', direction='ingress', ethertype='IPv4', id='531e856d-12ad-4575-bab3-d7f5e0afc16d', remote_ip_prefix='192.168.99.0/24', updated_at='2019-05-17T15:56:31Z' |
|                 | created_at='2019-05-17T15:56:30Z', direction='ingress', ethertype='IPv4', id='a21045f1-5c8a-4f7b-8611-1033614d4d6c', remote_ip_prefix='172.30.0.0/16', updated_at='2019-05-17T15:56:30Z'   |
|                 | created_at='2019-05-17T15:56:30Z', direction='egress', ethertype='IPv4', id='a287916d-a4aa-4c69-82a2-39fe21443d8a', updated_at='2019-05-17T15:56:30Z'                                      |
|                 | created_at='2019-05-17T15:56:30Z', direction='egress', ethertype='IPv6', id='fd65b869-7be1-4637-acd0-688f978e1505', updated_at='2019-05-17T15:56:30Z'                                      |
| updated_at      | 2019-05-17T15:56:31Z                                                                                                                                                                       |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

- Count the ports on the pool '10.11.0.0/16'
openstack port list | grep 10.11. | grep ACTIVE | wc -l
201

- Create new pods so new ports are created and see how much time do they need to go to ACTIVE status
[openshift@master-0 ~]$
oc new-project test-sg
oc run --image kuryr/demo test && date

openstack port list | grep 10.11. | grep ACTIVE | wc -l
206


It takes now between 10 and 20 seconds to create/add 5 ports to VM trunk, instead of minutes.

Comment 3 errata-xmlrpc 2019-06-26 09:08:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1605


Note You need to log in before you can comment on or make changes to this bug.