Bug 1353188 - ssh is not working from undercloud to created VM because of different MTU values
Summary: ssh is not working from undercloud to created VM because of different MTU values
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 9.0 (Mitaka)
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: async
: 9.0 (Mitaka)
Assignee: Brent Eagles
QA Contact: Toni Freger
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-06 12:58 UTC by GenadiC
Modified: 2016-08-24 12:56 UTC (History)
8 users (show)

Fixed In Version: openstack-neutron-8.1.2-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-24 12:56:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 333333 0 None None None 2016-07-26 18:14:12 UTC
Red Hat Product Errata RHBA-2016:1759 0 normal SHIPPED_LIVE openstack-neutron bug fix advisory 2016-08-24 16:50:04 UTC

Description GenadiC 2016-07-06 12:58:55 UTC
Description of problem:
Working on OSPD with DVR enabled, ssh from undercloud to VM fails as different MTU values are configured on VM (MTU  1400) and network (MTU 1350)

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
0. Creat
1. One option is to run on OSPD with DVR configured environment the automation neutron test: git/neutron/neutron/tests/tempest/scenario/test_basic.py - you will fail in the last part of the test (ssh), or put breakpoint before ssh and check ssh doesn't work because of different MTU values
2. Another option is to create OSPD environment with DVR enabled
3.Create a router, with internal and external network
4. Create security group to enable ingress ssh 
5. Create a VM server
6. Create and associate FIP for VM
7. Try to ssh the VM from the undercloud

Actual results:
SSH is not working as network is configured with 1350 MTU value and VMs interface is configured with 1400 MTU value

Expected results:
The same MTU value should be configured and ssh should work

Additional info:
Changing MTU value to 1350 on VM solves the problem
Changing value for global_physnet_mtu to 1350 on /etc/neutron/neutron.conf of the controller solves the problem as well

Comment 2 Brent Eagles 2016-07-06 13:47:21 UTC
There are some known inconsistencies with how OSPd configures MTU that we are working on resolving (see https://review.openstack.org/#/c/333333/). Given your description, I'm going to guess that the default template parameters for NeutronDnsmasqOptions and NeutronTenantMtu were used and you are using tunneling is being used? If my assumption is correct, you can try altering NeutronTenantMtu to increase the MTU for the network. Due to some unfortunate mappings of parameters, it can be a little difficult to optimize everything, but if you set NeutronTenantMtu to 1500 the network should be lifted to 1450 (1500 - 50 bytes for tunneling overhead) and the MTU being passed to the guest via DHCP will be 1400. 

You could also try setting the NeutronDnsmasqOptions to fit more nicely - e.g: dhcp-option-force=26,1450. 

Let me know if the above assumptions were correct and whether the suggested changes help.

Comment 3 GenadiC 2016-07-26 08:38:24 UTC
We set NeutronTenantMtu to 1450 (the default was 1400) and created setup again.
Indeed the network MTU was lifted to 1400 (was 1350 by default), but the MTU of the VM was lifted too - to 1450, so we got to the exactly same problem as before.

I am not sure if it is related but looking on qrouter on compute node we see that qr has mtu of 1400 and rfp has mtu of 1450

Comment 4 Brent Eagles 2016-07-26 13:00:21 UTC
Can you check the NeutronDnsmasqOptions to see if it is passing the MTU to the VMs via DHCP? If it is "dhcp-option-force=26,%MTU%" in the heat templates, you need to change it to 'disconnect' it from NeutronTenantMtu and set it lower than NeutronTenantMtu by 50 bytes or more. For example:

if NeutronTenantMtu = 1500

change

NeutronDnsmasqOptions to "dhcp-option-force=26,1450"

Comment 5 GenadiC 2016-07-27 08:30:47 UTC
Indeed after your proposed change of NeutronDnsmasqOptions I get the same MTU values on the network and on the VM

Comment 16 Toni Freger 2016-08-23 18:31:18 UTC
Verified on latest version openstack-neutron-8.1.2-4.el7ost.noarch

MTU on both, network and VM is 1450

VM:

$ ifconfig
eth0      Link encap:Ethernet  HWaddr FA:16:3E:8F:ED:53  
          inet addr:12.12.12.6  Bcast:12.12.12.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe8f:ed53/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:183 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:23322 (22.7 KiB)  TX bytes:18374 (17.9 KiB)

NETWORK:
neutron net-show 15a1b5e2-6113-457d-94fd-07c71a7dd2b8
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2016-08-23T18:18:52                  |
| description               |                                      |
| id                        | 15a1b5e2-6113-457d-94fd-07c71a7dd2b8 |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| mtu                       | 1450

Comment 18 errata-xmlrpc 2016-08-24 12:56:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1759.html


Note You need to log in before you can comment on or make changes to this bug.