Bug 1564346 - Strange behaviour of MTU values of the tun0 of the node where router is present
Summary: Strange behaviour of MTU values of the tun0 of the node where router is present
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.10.0
Assignee: Ben Bennett
QA Contact: Meng Bo
URL:
Whiteboard:
Depends On:
Blocks: 1634089
TreeView+ depends on / blocked
 
Reported: 2018-04-06 02:52 UTC by Miheer Salunke
Modified: 2018-12-28 02:50 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The MTU of the tun0 is not correct when there are no pods on the node. Consequence: None, other than the 'ip a' output is confusing. Fix: Set the mtu explicitly for the tun0 port in ovs. Result: 'ip a' shows the correct MTU when there are no pods on the node.
Clone Of:
: 1634089 (view as bug list)
Environment:
Last Closed: 2018-07-30 19:11:39 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:12:03 UTC
Origin (Github) 19372 None None None 2018-04-16 18:59:24 UTC

Comment 2 Meng Bo 2018-04-09 05:53:46 UTC
I think since the router is running in host network mode by default. It does not use the container network actually. And the packets will not be encapsulated by vxlan. So it will use the default interface's MTU.

Comment 3 Ben Bennett 2018-04-10 18:38:13 UTC
I've reproduced this.  It is weird, but harmless.

As Meng Bo noted the router runs with HostNetworking (by default) so it does not get attached to the OVS bridge.

When a pod veth gets attached to the OVS bridge the MTU on tun0 goes to the correct value.  When all pod veths are removed, it goes back to 1500.

I'm not sure why that behavior occurs yet, but since the value is correct when the bridge has pod attached to it it is harmless.

Comment 4 Aaron Conole 2018-04-10 21:34:19 UTC
Nowhere in the OvS code is the MTU set without a corresponding change to the appropriate row in the mtu_requested column in the Interfaces table.

I'm not sure what is setting this, but it isn't OvS.  Maybe docker?  Kubernetes?

Comment 5 Ben Bennett 2018-04-11 17:43:01 UTC
I can reproduce this when docker and openshift are not running.


# ip link add v1 type veth peer name v2 mtu 1450   
# ip l set v1 up
# ip l set v2 up

# ip a s v2
14: v2@v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 66:8d:17:c1:38:75 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::648d:17ff:fec1:3875/64 scope link 
       valid_lft forever preferred_lft forever

# ip a s tun0
6: tun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 6e:f3:eb:d2:52:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.128.0.1/23 brd 10.128.1.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::6cf3:ebff:fed2:52e5/64 scope link 
       valid_lft forever preferred_lft forever

# ovs-vsctl show
a5ec748f-4dd7-4933-b4d7-92ac39724f5d
    Bridge "br0"
        fail_mode: secure
        Port "br0"
            Interface "br0"
                type: internal
        Port "vxlan0"
            Interface "vxlan0"
                type: vxlan
                options: {key=flow, remote_ip=flow}
        Port "tun0"
            Interface "tun0"
                type: internal
    ovs_version: "2.8.1"

# ovs-vsctl add-port br0 v2

# ip a s tun0
6: tun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 6e:f3:eb:d2:52:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.128.0.1/23 brd 10.128.1.255 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::6cf3:ebff:fed2:52e5/64 scope link 
       valid_lft forever preferred_lft forever

Comment 6 Dan Williams 2018-04-13 18:08:23 UTC
(In reply to Aaron Conole from comment #4)
> Nowhere in the OvS code is the MTU set without a corresponding change to the
> appropriate row in the mtu_requested column in the Interfaces table.
> 
> I'm not sure what is setting this, but it isn't OvS.  Maybe docker? 
> Kubernetes?

Kernel part of OVS maybe?

Comment 7 Aaron Conole 2018-04-13 18:38:38 UTC
I didn't see anything in kernel.

I'm not sure what's happening - additional strange behaviors:

1. If we make the tun port using ip tuntap add ..., the MTU won't change

2. If we let OvS create it by setting up an internal port (which is how OpenShift works), then the mtu will be adjusted when adding / removing other ports.

3. If we set up the internal port with mtu_requested= then the MTU will be assigned the value in the mtu_requested field.

I'm at a loss to explain this just yet, but I'll investigate it further.

Comment 8 Ben Bennett 2018-04-16 18:59:25 UTC
Posted a PR to request the right MTU when the OVS bridge is created.  https://github.com/openshift/origin/pull/19372

Comment 9 openshift-github-bot 2018-04-19 21:41:31 UTC
Commit pushed to master at https://github.com/openshift/origin

https://github.com/openshift/origin/commit/9df7ca9edfa7ec002567a12c13e3c78dd0a61afc
Explicitly set the MTU on the tun0 interface

Before this, we did not set the MTU on the tun0 interface explicitly.
Since that interface is of type internal it is created and managed by
OVS.  Because it is managed by OVS, the MTU changes when pods are
added and removed from the bridge.  With a pod we see the desired MTU
on tun0, but when there are no pods, the MTU goes back to 1500.  This
is normally fine, but if there are packets above the MTU going between
nodes using their SDN addresses, then those packets will get dropped
when there are no pods using the SDN on one of those nodes.

The change is to request the correct MTU when we add the port to OVS.

Fixes bug 1564346 (https://bugzilla.redhat.com/show_bug.cgi?id=1564346)

Comment 10 Ben Bennett 2018-04-20 13:48:32 UTC
This has been resolved in 3.10, but there should be no need to backport.  The behavior is surprising, but harmless.

Comment 12 Hongan Li 2018-05-17 09:25:16 UTC
Verified in atomic-openshift-3.10.0-0.47.0.git.0.2fffa04.el7 and cannot reproduce the issue. The tun0 MTU using the configured value when no veth attached to OVS bridge.

OS: Red Hat Enterprise Linux Server release 7.5 (Maipo)
kernel: Linux ip-172-18-5-139.ec2.internal 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Comment 14 errata-xmlrpc 2018-07-30 19:11:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816


Note You need to log in before you can comment on or make changes to this bug.