Description of problem: After completing an OVS>OVN migration, attempting to reduce the MTU on a new neutron tenant network (1442 to 1400), the new MTU never gets applied. MTU was lowered by: (overcloud) [stack@joea-director14 ~]$ openstack network show tenant2 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2019-02-07T17:16:19Z | | description | | | dns_domain | | | id | df1f839c-958e-4984-8bbf-52cdfcfde6be | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1442 | | name | tenant2 | | port_security_enabled | True | | project_id | 093a314be2d24707a21fcb75a4bc65a6 | | provider:network_type | geneve | | provider:physical_network | None | | provider:segmentation_id | 32 | | qos_policy_id | None | | revision_number | 7 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 79957257-6aaa-4814-b36d-97b4098697f5 | | tags | | | updated_at | 2019-02-07T17:37:31Z | +---------------------------+--------------------------------------+ (overcloud) [stack@joea-director14 ~]$ openstack network set --mtu 1400 tenant2 (overcloud) [stack@joea-director14 ~]$ openstack network show tenant2 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2019-02-07T17:16:19Z | | description | | | dns_domain | | | id | df1f839c-958e-4984-8bbf-52cdfcfde6be | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1400 | | name | tenant2 | | port_security_enabled | True | | project_id | 093a314be2d24707a21fcb75a4bc65a6 | | provider:network_type | geneve | | provider:physical_network | None | | provider:segmentation_id | 32 | | qos_policy_id | None | | revision_number | 7 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 79957257-6aaa-4814-b36d-97b4098697f5 | | tags | | | updated_at | 2019-02-07T17:37:31Z | +---------------------------+--------------------------------------+ I tried restarting all ovn and neutron containers, as well as services running on the host OS on controllers and compute nodes, but nothing has made this update. I've been verifying by restarting rhel7.6 instances on the tenant network, MTU received via DHCP has remained 1442. Version-Release number of selected component (if applicable): OSP14 How reproducible: Easily Steps to Reproduce: 1. Deploy OSP14 with OVS, perform OVS>OVN migration 2. Create new tenant networks (geneve) and start instances on this network 3. Modify neutron MTU for network down to 1400, restart controllers and compute nodes, start instances again, see that MTU from DHCP is still 1442. Actual results: MTU does not change Expected results: MTU should change -- instances should be informed of 1400 MTU after restart Additional info: OSP14 Hackfest
I confirm this is a bug :) [centos@centos ~]$ openstack network set --mtu 1400 tenant1 [centos@centos ~]$ sudo ovn-nbctl list dhcp_options _uuid : 42089f8b-b1f7-4585-829e-4e90c82a292e cidr : "20.0.0.0/24" external_ids : {"neutron:revision_number"="0", subnet_id="4dbceb38-fa80-4af5-8916-643e7c8a8d1c"} options : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1442", router="20.0.0.1", server_id="20.0.0.1", server_mac="fa:16:3e:c8:a5:64"}
It has nothing to do with
This has nothing to do with the migration, hence, changing the title of the BZ. Currently networking-ovn won't honor any MTU changes to the network. The reason this is happening is that networking-ovn maps subnets to OVN DHCP_Options rows (where we store the MTU). As, from API perspective, the MTU is applied per network, updating the network won't get all it subnets updated, hence the MTU change is not applied to the DHCP_Options table. I could confirm it by just updating the subnet by changing the name for example: [centos@centos ~]$ openstack network set --mtu 1400 tenant1 [centos@centos ~]$ sudo ovn-nbctl list dhcp_options _uuid : 42089f8b-b1f7-4585-829e-4e90c82a292e cidr : "20.0.0.0/24" external_ids : {"neutron:revision_number"="0", subnet_id="4dbceb38-fa80-4af5-8916-643e7c8a8d1c"} options : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1442", router="20.0.0.1", server_id="20.0.0.1", server_mac="fa:16:3e:c8:a5:64"} [centos@centos networking-ovn]$ sudo ovn-nbctl list dhcp_options _uuid : b29962c9-2a78-4ae7-b243-d5d516b02ec2 cidr : "172.24.4.0/24" external_ids : {"neutron:revision_number"="0", subnet_id="5f7fa556-1d2b-4fca-b2fa-91202c3b3d74"} options : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1442", router="172.24.4.1", server_id="172.24.4.1", server_mac="fa:16:3e:92:f1:81"} [centos@centos networking-ovn]$ openstack subnet set --name subnet11 subnet1 _uuid : 42089f8b-b1f7-4585-829e-4e90c82a292e cidr : "20.0.0.0/24" external_ids : {"neutron:revision_number"="1", subnet_id="4dbceb38-fa80-4af5-8916-643e7c8a8d1c"} options : {dns_server="{10.200.16.3, 10.200.16.2, 10.200.16.4}", lease_time="43200", mtu="1400", router="20.0.0.1", server_id="20.0.0.1", server_mac="fa:16:3e:c8:a5:64"} The fix should be easy: when updating a network [0], if the MTU changed, update all subnets within that network as well. And add testing coverage. [0] https://github.com/openstack/networking-ovn/blob/0cf3e6643af5e45ae62ed26d009e892a20dabd7a/networking_ovn/common/ovn_client.py#L1376
Thanks for looking at this. I did poke at this issue a bit more after opening the bug, and I further observed that after restarting all controllers, computes, and instances, L2 transport for the tenant network is using the new lower MTU (1400), but DHCP option is still sending the original higher MTU (1442), so I wound up with packet loss at packets larger than 1400 bytes.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3750