In case when on one compute node there is many ports which are using same security group and this security group uses rule to allow traffic from ports with given SG adding new port to host is very slow, as all the related ports needs to be taken into account. We tested it on compute with 170 trunk supports and it took about 5 minutes for new port to become ACTIVE. In order to avoid this problem until [1] is solved, there is a need for moving away from remote_group_id when possible, in favor of remote_prefix_ip [1] https://bugzilla.redhat.com/show_bug.cgi?id=1703467
Verified in openshift-ansible-3.11.115-1 on top of OSP13 2019-05-15.1 puddle. Verification steps: - Deploy OSP 13 with Octavia and ML2/OVS Neutron backend - Deploy OCP on top with Kuryr SDN, and without namespace isolation - Create 150 pods (in 3 different projects) so many ports are created [openshift@master-0 ~]$ oc new-project test oc run --image kuryr/demo demo oc scale dc/demo --replicas=50 oc new-project test2 oc run --image kuryr/demo demo oc scale dc/demo --replicas=50 oc new-project test3 oc run --image kuryr/demo demo oc scale dc/demo --replicas=50 oc get pods --all-namespaces | grep test | grep Running | wc -l 150 - Check the SG for pod/service is using remote_ip_prefix (shiftstack) [cloud-user@ansible-host-0 ~]$ openstack security group list +--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+ | ID | Name | Description | Project | +--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+ | 45468573-3b83-44a8-af7d-ab1f35a65fd6 | openshift-ansible-openshift.example.com-infra-secgrp | Security group for openshift.example.com OpenShift infrastructure cluster nodes | bad742dd55e64ee1a2f7fdd2f0818bad | | 4610f3f8-bbe9-4d1c-99eb-dc2d7982b80a | openshift-ansible-openshift.example.com-pod-service-secgrp | Give services and nodes access to the pods | bad742dd55e64ee1a2f7fdd2f0818bad | | 56173aac-d734-4735-9d5d-ff68dfe8894e | default | Default security group | bad742dd55e64ee1a2f7fdd2f0818bad | | 5746452e-a079-4254-90bb-f02bacad8d43 | openshift-ansible-openshift.example.com-node-secgrp | Security group for openshift.example.com OpenShift cluster nodes | bad742dd55e64ee1a2f7fdd2f0818bad | | 734b36b0-1e54-4dda-ab3c-1f7765db0aac | openshift-ansible-openshift.example.com-lb-secgrp | Security group for openshift.example.com cluster Load Balancer | bad742dd55e64ee1a2f7fdd2f0818bad | | a1267a46-826e-40de-8261-a3ac16b9b4ae | openshift-ansible-openshift.example.com-etcd-secgrp | Security group for openshift.example.com etcd cluster | bad742dd55e64ee1a2f7fdd2f0818bad | | a510e6ee-085c-4aeb-9a21-db75f1692bae | openshift-ansible-openshift.example.com-common-secgrp | Basic ssh/icmp security group for openshift.example.com OpenShift cluster | bad742dd55e64ee1a2f7fdd2f0818bad | | aeab021c-14ff-4a27-aac8-54e741a2aaac | openshift-ansible-openshift.example.com-master-secgrp | Security group for openshift.example.com OpenShift cluster master | bad742dd55e64ee1a2f7fdd2f0818bad | | dcd9cafc-3d6b-4aee-b524-8ce5a18e263a | secgroup_openshift_dns | ir: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-osasinfra-shiftstack_on_vms-13-customized-job-hybrid/82/ at 2019-05-17T14:58:05Z | bad742dd55e64ee1a2f7fdd2f0818bad | | feacb4d7-9b38-4df0-870c-806fef55990f | secgroup_openshift | ir: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-osasinfra-shiftstack_on_vms-13-customized-job-hybrid/82/ at 2019-05-17T14:58:05Z | bad742dd55e64ee1a2f7fdd2f0818bad | +--------------------------------------+------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+ (shiftstack) [cloud-user@ansible-host-0 ~]$ openstack security group show openshift-ansible-openshift.example.com-pod-service-secgrp +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | created_at | 2019-05-17T15:56:30Z | | description | Give services and nodes access to the pods | | id | 4610f3f8-bbe9-4d1c-99eb-dc2d7982b80a | | name | openshift-ansible-openshift.example.com-pod-service-secgrp | | project_id | bad742dd55e64ee1a2f7fdd2f0818bad | | revision_number | 5 | | rules | created_at='2019-05-17T15:56:31Z', direction='ingress', ethertype='IPv4', id='2e0324a4-f1bc-4ed4-b744-7ad7787147f8', remote_ip_prefix='10.11.0.0/16', updated_at='2019-05-17T15:56:31Z' | | | created_at='2019-05-17T15:56:31Z', direction='ingress', ethertype='IPv4', id='531e856d-12ad-4575-bab3-d7f5e0afc16d', remote_ip_prefix='192.168.99.0/24', updated_at='2019-05-17T15:56:31Z' | | | created_at='2019-05-17T15:56:30Z', direction='ingress', ethertype='IPv4', id='a21045f1-5c8a-4f7b-8611-1033614d4d6c', remote_ip_prefix='172.30.0.0/16', updated_at='2019-05-17T15:56:30Z' | | | created_at='2019-05-17T15:56:30Z', direction='egress', ethertype='IPv4', id='a287916d-a4aa-4c69-82a2-39fe21443d8a', updated_at='2019-05-17T15:56:30Z' | | | created_at='2019-05-17T15:56:30Z', direction='egress', ethertype='IPv6', id='fd65b869-7be1-4637-acd0-688f978e1505', updated_at='2019-05-17T15:56:30Z' | | updated_at | 2019-05-17T15:56:31Z | +-----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - Count the ports on the pool '10.11.0.0/16' openstack port list | grep 10.11. | grep ACTIVE | wc -l 201 - Create new pods so new ports are created and see how much time do they need to go to ACTIVE status [openshift@master-0 ~]$ oc new-project test-sg oc run --image kuryr/demo test && date openstack port list | grep 10.11. | grep ACTIVE | wc -l 206 It takes now between 10 and 20 seconds to create/add 5 ports to VM trunk, instead of minutes.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1605