Bug 1799078

Summary:	Cannot specify MTU of OpenShift network in OpenStack IPI
Product:	OpenShift Container Platform	Reporter:	Andreas Karis <akaris>
Component:	Installer	Assignee:	Eric Duen <eduen>
Installer sub component:	OpenShift on OpenStack	QA Contact:	David Sanz <dsanzmor>
Status:	CLOSED NOTABUG	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	asimonel, hgomes, jlema, pprinett, racedoro, tsedovic
Version:	4.3.z
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-14 15:25:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andreas Karis 2020-02-06 15:25:15 UTC

Description of problem:

Hi,

With OCP 4.3, we have no way to specify the MTU of the neutron network that is generated by terraform. Neutron allows administrators to generate networks with the --mtu parameter. It seems that the OpenShift installer either doesn't set this value (and uses the OSP default) or uses the maximum MTU (somehow). In either case, administrator need to be able to provide the MTU for the neutron network in install-config.yaml

overcloud) [root@openshift-bootstrap-vm ~]# neutron net-show 302e4ce7-19ab-49df-94d1-68854ca12eb5
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead:
~~~
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2020-02-06T09:55:32Z                 |
| description               |                                      |
| id                        | 302e4ce7-19ab-49df-94d1-68854ca12eb5 |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| l2_adjacency              | True                                 |
| mtu                       | 8950                                 |
| name                      | cluster02-ctjd4-openshift            |
| port_security_enabled     | True                                 |
| project_id                | 218d8aea39b54baabc3726ebabeaa22d     |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 10103                                |
| qos_policy_id             |                                      |
| revision_number           | 5                                    |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 2f5015a9-3c94-49e4-af0d-b62feebec13c |
| tags                      | openshiftClusterID=cluster02-ctjd4   |
| tenant_id                 | 218d8aea39b54baabc3726ebabeaa22d     |
| updated_at                | 2020-02-06T09:55:38Z                 |
+---------------------------+--------------------------------------+
~~~

This is critical when there are MTU mismatches elsewhere in the environment. In the attached customer case, we needed to work around this issue (that's outside of the realm of control of OpenShift) by running the following just before the nova instances spawn:
~~~
openstack network set 302e4ce7-19ab-49df-94d1-68854ca12eb5 --mtu=1450
~~~

It doesn't matter if the problem lies within OpenStack or within the physical network: the OpenShift installer should/must provide and option so that we can configure the MTU of this network.


Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Andreas Karis 2020-02-07 08:04:17 UTC

We see from https://github.com/openshift/installer/blob/b87ca03305176def4bd0443ec1be96e01972d1ac/pkg/types/openstack/platform.go

That in OCP 4.3,  'externalDNS' was added to the configuration. We should add a parameter, 'networkMTU' (or whatever better naming) and request a specific MTU when creating the internal network.

Otherwise, at the customer site, we need to apply the following cumbersome workaround at the moment:
~~~
# Start OCP IPI installation and wait until networks are created and bootstrap node is up
# the OCP network now spawns with the max MTU in the cluster, but due to MTU mismatch issues in the path, traffic is
# being blackholed
openstack network set --mtu 1450 <openshift internal network>
Go to bootstrap node console in horizon and send <CTRL-ALT-DEL> 
~~~

- Andreas

Comment 4 Tomas Sedovic 2020-02-13 15:17:35 UTC

The OpenShift IPI is generally against adding extra configuration options. Adding more individual options is generally not going to be well received by the OpenShift project.

Currently, the only options are going full UPI or changing this option using Neutron manually on the fly.

But we are looking at implementing bring-your-own networking (a feature already available in the AWS provider). That is a hybrid between IPI and UPI where you prepare your networking the way you want, and then have the installer create the servers and set up everything else in the networks.

We propose to use this "Customer-provisioned Network & Subnets" feature for any specific Neutron configurations rather than adding more install-config knobs one by one.

Comment 6 Ramon Acedo 2020-02-14 09:55:52 UTC

Andreas many thanks for the detailed feedback. As Tomas explains, this problem will be solved by a future feature where you will be able to create your networking in OpenStack and then instruct the installer to use the existing networking resources while still installing with the IPI workflow. The upcoming UPI installation for OSP will also allow this.

Comment 7 Pierre Prinetti 2020-02-14 15:25:07 UTC

Closing; the linked feature is expected to supersede this request.