Bug 1703947

Summary: Using remote_group_id affects the subports attachment to the trunks
Product: OpenShift Container Platform Reporter: Luis Tomas Bolivar <ltomasbo>
Component: InstallerAssignee: Luis Tomas Bolivar <ltomasbo>
Installer sub component: openshift-ansible QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent    
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-26 09:08:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luis Tomas Bolivar 2019-04-29 07:53:45 UTC
In case when on one compute node there is many ports which are using same security group and this security group uses rule to allow traffic from ports with given SG adding new port to host is very slow, as all the related ports needs to be taken into account.

We tested it on compute with 170 trunk supports and it took about 5 minutes for new port to become ACTIVE.

In order to avoid this problem until [1] is solved, there is a need for moving away from remote_group_id when possible, in favor of remote_prefix_ip

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1703467

Comment 1 Jon Uriarte 2019-05-17 18:06:10 UTC
Verified in openshift-ansible-3.11.115-1 on top of OSP13 2019-05-15.1 puddle.

Verification steps:
- Deploy OSP 13 with Octavia and ML2/OVS Neutron backend
- Deploy OCP on top with Kuryr SDN, and without namespace isolation
- Create 150 pods (in 3 different projects) so many ports are created

[openshift@master-0 ~]$
oc new-project test
oc run --image kuryr/demo demo
oc scale dc/demo --replicas=50                                                                                                                                                             

oc new-project test2
oc run --image kuryr/demo demo                                                                                                                                                             
oc scale dc/demo --replicas=50                                                                                                                                                             

oc new-project test3
oc run --image kuryr/demo demo                                                                                                                                                             
oc scale dc/demo --replicas=50                                                                    

oc get pods --all-namespaces | grep test | grep Running | wc -l
150

- Check the SG for pod/service is using remote_ip_prefix
(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack security group list
+--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+
| ID                                   | Name                                                       | Description                                                                                                                                           | Project                          |
+--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+
| 45468573-3b83-44a8-af7d-ab1f35a65fd6 | openshift-ansible-openshift.example.com-infra-secgrp       | Security group for openshift.example.com OpenShift infrastructure cluster nodes                                                                       | bad742dd55e64ee1a2f7fdd2f0818bad |
| 4610f3f8-bbe9-4d1c-99eb-dc2d7982b80a | openshift-ansible-openshift.example.com-pod-service-secgrp | Give services and nodes access to the pods                                                                                                            | bad742dd55e64ee1a2f7fdd2f0818bad |
| 56173aac-d734-4735-9d5d-ff68dfe8894e | default                                                    | Default security group                                                                                                                                | bad742dd55e64ee1a2f7fdd2f0818bad |
| 5746452e-a079-4254-90bb-f02bacad8d43 | openshift-ansible-openshift.example.com-node-secgrp        | Security group for openshift.example.com OpenShift cluster nodes                                                                                      | bad742dd55e64ee1a2f7fdd2f0818bad |
| 734b36b0-1e54-4dda-ab3c-1f7765db0aac | openshift-ansible-openshift.example.com-lb-secgrp          | Security group for openshift.example.com cluster Load Balancer                                                                                        | bad742dd55e64ee1a2f7fdd2f0818bad |
| a1267a46-826e-40de-8261-a3ac16b9b4ae | openshift-ansible-openshift.example.com-etcd-secgrp        | Security group for openshift.example.com etcd cluster                                                                                                 | bad742dd55e64ee1a2f7fdd2f0818bad |
| a510e6ee-085c-4aeb-9a21-db75f1692bae | openshift-ansible-openshift.example.com-common-secgrp      | Basic ssh/icmp security group for openshift.example.com OpenShift cluster                                                                             | bad742dd55e64ee1a2f7fdd2f0818bad |
| aeab021c-14ff-4a27-aac8-54e741a2aaac | openshift-ansible-openshift.example.com-master-secgrp      | Security group for openshift.example.com OpenShift cluster master                                                                                     | bad742dd55e64ee1a2f7fdd2f0818bad |
| dcd9cafc-3d6b-4aee-b524-8ce5a18e263a | secgroup_openshift_dns                                     | ir: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-osasinfra-shiftstack_on_vms-13-customized-job-hybrid/82/ at  2019-05-17T14:58:05Z | bad742dd55e64ee1a2f7fdd2f0818bad |
| feacb4d7-9b38-4df0-870c-806fef55990f | secgroup_openshift                                         | ir: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-osasinfra-shiftstack_on_vms-13-customized-job-hybrid/82/ at  2019-05-17T14:58:05Z | bad742dd55e64ee1a2f7fdd2f0818bad |
+--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+


(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack security group show openshift-ansible-openshift.example.com-pod-service-secgrp
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                      |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at      | 2019-05-17T15:56:30Z                                                                                                                                                                       |
| description     | Give services and nodes access to the pods                                                                                                                                                 |
| id              | 4610f3f8-bbe9-4d1c-99eb-dc2d7982b80a                                                                                                                                                       |
| name            | openshift-ansible-openshift.example.com-pod-service-secgrp                                                                                                                                 |
| project_id      | bad742dd55e64ee1a2f7fdd2f0818bad                                                                                                                                                           |
| revision_number | 5                                                                                                                                                                                          |
| rules           | created_at='2019-05-17T15:56:31Z', direction='ingress', ethertype='IPv4', id='2e0324a4-f1bc-4ed4-b744-7ad7787147f8', remote_ip_prefix='10.11.0.0/16', updated_at='2019-05-17T15:56:31Z'    |
|                 | created_at='2019-05-17T15:56:31Z', direction='ingress', ethertype='IPv4', id='531e856d-12ad-4575-bab3-d7f5e0afc16d', remote_ip_prefix='192.168.99.0/24', updated_at='2019-05-17T15:56:31Z' |
|                 | created_at='2019-05-17T15:56:30Z', direction='ingress', ethertype='IPv4', id='a21045f1-5c8a-4f7b-8611-1033614d4d6c', remote_ip_prefix='172.30.0.0/16', updated_at='2019-05-17T15:56:30Z'   |
|                 | created_at='2019-05-17T15:56:30Z', direction='egress', ethertype='IPv4', id='a287916d-a4aa-4c69-82a2-39fe21443d8a', updated_at='2019-05-17T15:56:30Z'                                      |
|                 | created_at='2019-05-17T15:56:30Z', direction='egress', ethertype='IPv6', id='fd65b869-7be1-4637-acd0-688f978e1505', updated_at='2019-05-17T15:56:30Z'                                      |
| updated_at      | 2019-05-17T15:56:31Z                                                                                                                                                                       |
+-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

- Count the ports on the pool '10.11.0.0/16'
openstack port list | grep 10.11. | grep ACTIVE | wc -l
201

- Create new pods so new ports are created and see how much time do they need to go to ACTIVE status
[openshift@master-0 ~]$
oc new-project test-sg
oc run --image kuryr/demo test && date

openstack port list | grep 10.11. | grep ACTIVE | wc -l
206


It takes now between 10 and 20 seconds to create/add 5 ports to VM trunk, instead of minutes.

Comment 3 errata-xmlrpc 2019-06-26 09:08:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1605