Bug 1353188

Summary: ssh is not working from undercloud to created VM because of different MTU values
Product: Red Hat OpenStack Reporter: GenadiC <gcheresh>
Component: openstack-neutronAssignee: Brent Eagles <beagles>
Status: CLOSED ERRATA QA Contact: Toni Freger <tfreger>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: amuller, beagles, chrisw, gcheresh, nyechiel, oblaut, sclewis, srevivo
Target Milestone: asyncKeywords: ZStream
Target Release: 9.0 (Mitaka)   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-8.1.2-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-24 12:56:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description GenadiC 2016-07-06 12:58:55 UTC
Description of problem:
Working on OSPD with DVR enabled, ssh from undercloud to VM fails as different MTU values are configured on VM (MTU  1400) and network (MTU 1350)

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
0. Creat
1. One option is to run on OSPD with DVR configured environment the automation neutron test: git/neutron/neutron/tests/tempest/scenario/test_basic.py - you will fail in the last part of the test (ssh), or put breakpoint before ssh and check ssh doesn't work because of different MTU values
2. Another option is to create OSPD environment with DVR enabled
3.Create a router, with internal and external network
4. Create security group to enable ingress ssh 
5. Create a VM server
6. Create and associate FIP for VM
7. Try to ssh the VM from the undercloud

Actual results:
SSH is not working as network is configured with 1350 MTU value and VMs interface is configured with 1400 MTU value

Expected results:
The same MTU value should be configured and ssh should work

Additional info:
Changing MTU value to 1350 on VM solves the problem
Changing value for global_physnet_mtu to 1350 on /etc/neutron/neutron.conf of the controller solves the problem as well

Comment 2 Brent Eagles 2016-07-06 13:47:21 UTC
There are some known inconsistencies with how OSPd configures MTU that we are working on resolving (see https://review.openstack.org/#/c/333333/). Given your description, I'm going to guess that the default template parameters for NeutronDnsmasqOptions and NeutronTenantMtu were used and you are using tunneling is being used? If my assumption is correct, you can try altering NeutronTenantMtu to increase the MTU for the network. Due to some unfortunate mappings of parameters, it can be a little difficult to optimize everything, but if you set NeutronTenantMtu to 1500 the network should be lifted to 1450 (1500 - 50 bytes for tunneling overhead) and the MTU being passed to the guest via DHCP will be 1400. 

You could also try setting the NeutronDnsmasqOptions to fit more nicely - e.g: dhcp-option-force=26,1450. 

Let me know if the above assumptions were correct and whether the suggested changes help.

Comment 3 GenadiC 2016-07-26 08:38:24 UTC
We set NeutronTenantMtu to 1450 (the default was 1400) and created setup again.
Indeed the network MTU was lifted to 1400 (was 1350 by default), but the MTU of the VM was lifted too - to 1450, so we got to the exactly same problem as before.

I am not sure if it is related but looking on qrouter on compute node we see that qr has mtu of 1400 and rfp has mtu of 1450

Comment 4 Brent Eagles 2016-07-26 13:00:21 UTC
Can you check the NeutronDnsmasqOptions to see if it is passing the MTU to the VMs via DHCP? If it is "dhcp-option-force=26,%MTU%" in the heat templates, you need to change it to 'disconnect' it from NeutronTenantMtu and set it lower than NeutronTenantMtu by 50 bytes or more. For example:

if NeutronTenantMtu = 1500

change

NeutronDnsmasqOptions to "dhcp-option-force=26,1450"

Comment 5 GenadiC 2016-07-27 08:30:47 UTC
Indeed after your proposed change of NeutronDnsmasqOptions I get the same MTU values on the network and on the VM

Comment 16 Toni Freger 2016-08-23 18:31:18 UTC
Verified on latest version openstack-neutron-8.1.2-4.el7ost.noarch

MTU on both, network and VM is 1450

VM:

$ ifconfig
eth0      Link encap:Ethernet  HWaddr FA:16:3E:8F:ED:53  
          inet addr:12.12.12.6  Bcast:12.12.12.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe8f:ed53/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:183 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:23322 (22.7 KiB)  TX bytes:18374 (17.9 KiB)

NETWORK:
neutron net-show 15a1b5e2-6113-457d-94fd-07c71a7dd2b8
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        | nova                                 |
| created_at                | 2016-08-23T18:18:52                  |
| description               |                                      |
| id                        | 15a1b5e2-6113-457d-94fd-07c71a7dd2b8 |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| mtu                       | 1450

Comment 18 errata-xmlrpc 2016-08-24 12:56:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1759.html