Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1673690

Summary:	OSP14 Hackfest: OVN Neutron network MTU never get applied
Product:	Red Hat OpenStack	Reporter:	Joe Antkowiak <joea>
Component:	python-networking-ovn	Assignee:	Kamil Sambor <ksambor>
Status:	CLOSED ERRATA	QA Contact:	Eran Kuris <ekuris>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	14.0 (Rocky)	CC:	apevec, dalvarez, lhh, lmartins, majopela
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	python-networking-ovn-5.0.2-0.20190430191341.e673daf.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-06 16:50:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joe Antkowiak 2019-02-07 19:11:50 UTC

Description of problem:
After completing an OVS>OVN migration, attempting to reduce the MTU on a new neutron tenant network (1442 to 1400), the new MTU never gets applied.

MTU was lowered by:

(overcloud) [stack@joea-director14 ~]$ openstack network show tenant2
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2019-02-07T17:16:19Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | df1f839c-958e-4984-8bbf-52cdfcfde6be |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1442                                 |
| name                      | tenant2                              |
| port_security_enabled     | True                                 |
| project_id                | 093a314be2d24707a21fcb75a4bc65a6     |
| provider:network_type     | geneve                               |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 32                                   |
| qos_policy_id             | None                                 |
| revision_number           | 7                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 79957257-6aaa-4814-b36d-97b4098697f5 |
| tags                      |                                      |
| updated_at                | 2019-02-07T17:37:31Z                 |
+---------------------------+--------------------------------------+
(overcloud) [stack@joea-director14 ~]$ openstack network set --mtu 1400 tenant2                                                                                                                                  
(overcloud) [stack@joea-director14 ~]$ openstack network show tenant2
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2019-02-07T17:16:19Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | df1f839c-958e-4984-8bbf-52cdfcfde6be |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | 1400                                 |
| name                      | tenant2                              |
| port_security_enabled     | True                                 |
| project_id                | 093a314be2d24707a21fcb75a4bc65a6     |
| provider:network_type     | geneve                               |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 32                                   |
| qos_policy_id             | None                                 |
| revision_number           | 7                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 79957257-6aaa-4814-b36d-97b4098697f5 |
| tags                      |                                      |
| updated_at                | 2019-02-07T17:37:31Z                 |
+---------------------------+--------------------------------------+
 

I tried restarting all ovn and neutron containers, as well as services running on the host OS on controllers and compute nodes, but nothing has made this update.

I've been verifying by restarting rhel7.6 instances on the tenant network, MTU received via DHCP has remained 1442.


Version-Release number of selected component (if applicable):

OSP14

How reproducible:

Easily

Steps to Reproduce:
1. Deploy OSP14 with OVS, perform OVS>OVN migration
2. Create new tenant networks (geneve) and start instances on this network
3. Modify neutron MTU for network down to 1400, restart controllers and compute nodes, start instances again, see that MTU from DHCP is still 1442.

Actual results:

MTU does not change

Expected results:

MTU should change -- instances should be informed of 1400 MTU after restart


Additional info:
OSP14 Hackfest

Comment 1 Daniel Alvarez Sanchez 2019-02-08 16:13:40 UTC

I confirm this is a bug :)

[centos@centos ~]$ openstack network set --mtu 1400 tenant1
[centos@centos ~]$ sudo ovn-nbctl list dhcp_options
_uuid               : 42089f8b-b1f7-4585-829e-4e90c82a292e
cidr                : "20.0.0.0/24"
external_ids        : {"neutron:revision_number"="0", subnet_id="4dbceb38-fa80-4af5-8916-643e7c8a8d1c"}
options             : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1442", router="20.0.0.1", server_id="20.0.0.1", server_mac="fa:16:3e:c8:a5:64"}

Comment 2 Daniel Alvarez Sanchez 2019-02-08 16:22:09 UTC

It has nothing to do with

Comment 3 Daniel Alvarez Sanchez 2019-02-08 16:32:38 UTC

This has nothing to do with the migration, hence, changing the title of the BZ.

Currently networking-ovn won't honor any MTU changes to the network.
The reason this is happening is that networking-ovn maps subnets to OVN DHCP_Options rows (where we store the MTU). As, from API perspective, the MTU is applied per network, updating the network won't get all it subnets updated, hence the MTU change is not applied to the DHCP_Options table.

I could confirm it by just updating the subnet by changing the name for example:


[centos@centos ~]$ openstack network set --mtu 1400 tenant1
[centos@centos ~]$ sudo ovn-nbctl list dhcp_options
_uuid               : 42089f8b-b1f7-4585-829e-4e90c82a292e
cidr                : "20.0.0.0/24"
external_ids        : {"neutron:revision_number"="0", subnet_id="4dbceb38-fa80-4af5-8916-643e7c8a8d1c"}
options             : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1442", router="20.0.0.1", server_id="20.0.0.1", server_mac="fa:16:3e:c8:a5:64"}

[centos@centos networking-ovn]$ sudo ovn-nbctl list dhcp_options
_uuid               : b29962c9-2a78-4ae7-b243-d5d516b02ec2
cidr                : "172.24.4.0/24"
external_ids        : {"neutron:revision_number"="0", subnet_id="5f7fa556-1d2b-4fca-b2fa-91202c3b3d74"}
options             : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1442", router="172.24.4.1", server_id="172.24.4.1", server_mac="fa:16:3e:92:f1:81"}

[centos@centos networking-ovn]$ openstack subnet set --name subnet11 subnet1
_uuid               : 42089f8b-b1f7-4585-829e-4e90c82a292e
cidr                : "20.0.0.0/24"
external_ids        : {"neutron:revision_number"="1", subnet_id="4dbceb38-fa80-4af5-8916-643e7c8a8d1c"}
options             : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1400", router="20.0.0.1", server_id="20.0.0.1", server_mac="fa:16:3e:c8:a5:64"}


The fix should be easy: when updating a network [0], if the MTU changed, update all subnets within that network as well.
And add testing coverage.

[0] https://github.com/openstack/networking-ovn/blob/0cf3e6643af5e45ae62ed26d009e892a20dabd7a/networking_ovn/common/ovn_client.py#L1376

Comment 4 Joe Antkowiak 2019-02-08 17:12:42 UTC

Thanks for looking at this.

I did poke at this issue a bit more after opening the bug, and I further observed that after restarting all controllers, computes, and instances, L2 transport for the tenant network is using the new lower MTU (1400), but DHCP option is still sending the original higher MTU (1442), so I wound up with packet loss at packets larger than 1400 bytes.

Comment 7 errata-xmlrpc 2019-11-06 16:50:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3750