Bug 1824287 - Using security rules with remote security group may cause deployment to fail
Summary: Using security rules with remote security group may cause deployment to fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.5.0
Assignee: Martin André
QA Contact: David Sanz
URL:
Whiteboard:
Depends On:
Blocks: 1825286 1825460
TreeView+ depends on / blocked
 
Reported: 2020-04-15 17:53 UTC by Adolfo Duarte
Modified: 2020-07-13 17:28 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The openstack IPI installer creates Security groups using `remote_group_id` to allow traffic origins. Consequence: Using `remote_group_id` in the security rules is very inefficient, triggering a lot of computation by ovs agent to generate the flows and possibly exceeding the time allocated for flow generation. In such cases, especially in environments already under stress, masters nodes may be unable to communicate with worker nodes, leading the deployment to fail. Fix: Use IP prefixes for whitelisting traffic origins instead of `remote_group_id` Result: Less load on Neutron resources should reduce the occurrence of timeouts
Clone Of:
: 1825286 1825460 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:27:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift installer pull 3461 None closed Bug 1824287: OpenStack: Replace remote_group_id with remote_ip_prefix 2020-07-22 12:55:33 UTC
Red Hat Product Errata RHBA-2020:2409 None None None 2020-07-13 17:28:28 UTC

Description Adolfo Duarte 2020-04-15 17:53:34 UTC
Using security groups as the destination or source of a security rule on openstack is very resource intensive. This can lead to network traffic performance issues with openstack neutron. 
The degraded network traffic can lead to installation failure where the bootstrap process times out because pods can access resources through the openshift sdn internal network.  
For example, some pods are unable to succesfully resolv ip addresses because they can't reach the internal dns service of the cluster. 

Communication between pods is spotty and leads to cascade failures.

Comment 1 Martin André 2020-04-16 07:50:21 UTC
Using `remote_group_id` in the security rules is very inefficient, triggering a lot of computation by ovs agent to generate the flows and possibly exceeding the time allocated for flow generation. In such cases, especially in environments already under stress, masters nodes may be unable to communicate with worker nodes, leading the deployment to fail.

We're seeing this behavior in MOC, the cloud we're using for our CI.

The workaround is to use the more efficient remote_ip_prefix rather than remote_group_id when creating security rules.

This was already done for openshift-ansible in the past: https://bugzilla.redhat.com/show_bug.cgi?id=1703947

Comment 4 Pierre Prinetti 2020-04-20 08:11:34 UTC
A note for the verifier QE.

This bug affects our CI. As a result, we can already prove the effectiveness of the patch: jobs are green again after the merge.

We would still need your help for the usual regression / edge case testing.

Thank you!

Comment 5 David Sanz 2020-04-20 11:00:17 UTC
No failure detected on latest 4.5 nightly after patch is merged, and secgroup rules are fine.

Marking as verified

Comment 6 errata-xmlrpc 2020-07-13 17:27:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.