Bug 1475460 - Path MTU discovery not working when run from OpenStack
Summary: Path MTU discovery not working when run from OpenStack
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: docker
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Daniel Walsh
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-26 18:03 UTC by Jonathan Lebon
Modified: 2018-05-29 11:59 UTC (History)
19 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2018-05-29 11:59:32 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Jonathan Lebon 2017-07-26 18:03:08 UTC
Description of problem:

In https://github.com/moby/moby/pull/13953, docker was changed to inherit the MTU from the default route.

Later in https://github.com/moby/moby/pull/18108, docker was changed back to *not* inherit from the default route and just let the kernel figure out the optimal connection MTU using path MTU discovery.

However, I'm observing here that this is still not working and that one essentially has to use the --mtu option.

E.g. in an OpenStack VM running the latest F26 AH:

# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
<snip>
# docker run --rm registry.fedoraproject.org/fedora:26 sh -c 'dnf install -y ip route && ip addr'
<snip>
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
<snip>

This mismatch is fine, and IIUC in https://github.com/moby/moby/pull/18108, the kernel should be able to figure it out. But that doesn't seem to happen and as a result, we get severe network performance issues. So there's something wrong along the networking stack: either it's something in the kernel (e.g. a regression from https://bugzilla.redhat.com/show_bug.cgi?id=1441310 perhaps?), or there's something in docker preventing the kernel from doing its job, or there something wonky in OpenStack in the way it handles packages larger than $MTU.

Version-Release number of selected component (if applicable):

# rpm-ostree status
State: idle
Deployments:
* fedora-atomic:fedora/26/x86_64/updates/atomic-host
                Version: 26.91 (2017-07-23 03:12:08)
                 Commit: 0715ce81064c30d34ed52ef811a3ad5e5d6a34da980bf35b19312489b32d9b83
           GPGSignature: 1 signature
                         Signature made Sun Jul 23 03:13:09 2017 using RSA key ID 812A6B4B64DAB85D
                         Good signature from "Fedora 26 Primary <fedora-26-primary>"
# rpm -q docker kernel
docker-1.13.1-19.git27e468e.fc26.x86_64
kernel-4.11.11-300.fc26.x86_64

How reproducible:

Always

Steps to Reproduce:
1. Boot F26 AH in OpenStack
2. Start docker
3. Start a container
4. Try to curl something from an internal site, e.g.: curl -L http://download.devel.redhat.com/pub/fedora/linux/updates/26/x86_64/repodata/repomd.xml

Actual results:

curl invocation hangs

Expected results:

curl invocation works

Additional info:

As expected, changing options to e.g. --mtu 1400 allows the curl request to go through (or alternatively, creating the container with --net=host).

Comment 2 Colin Walters 2017-07-26 18:36:30 UTC
This looks useful:
https://blog.cloudflare.com/path-mtu-discovery-in-practice/

Comment 3 Colin Walters 2017-07-26 18:49:56 UTC
It looks like in this:
https://github.com/projectcalico/calicoctl/issues/488

They discussed ensuring the MTU isn't *larger* than the host routes, which makes sense to me.

Also found:
https://github.com/openshift/openshift-ansible/pull/544/commits/c28aea2f19b6aea729f27b636e5318ac775addb6
which looks to me like it hardcodes a default of 1450?

Comment 4 Micah Abbott 2017-07-26 19:20:47 UTC
This could possibly go back as far as docker 1.10.  We've been hitting this for a long time.

See the following, which I believe are all related.

https://bugzilla.redhat.com/show_bug.cgi?id=1333961

https://bugzilla.redhat.com/show_bug.cgi?id=1357116

https://projects.engineering.redhat.com/browse/CENTRALCI-1749

http://post-office.corp.redhat.com/archives/ci-infra-list/2017-March/msg00036.html

Comment 5 Karel Srot 2017-07-27 13:53:45 UTC
*** Bug 1474793 has been marked as a duplicate of this bug. ***

Comment 7 Sean Dilda 2017-09-21 02:09:18 UTC
I see this constantly on Fedora 26 with docker or with firejail.

In both cases I see the issue when I have a network namespace (from docker or firejail) connected to a bridged network interface and have an openconnect VPN session open.

In these cases all MTUs are 1500, except vpn0 (from openconnect) with an MTU of 1274.

Traffic from outside a namespace is fine, going through the vpn or not.  Traffic from inside the name space that doesn't go oever the VPN is fine.  But when it goes over the VPN, tcpdump shows large response packets are getting sent by remote servers, then dropped by vpn0.

I was able to work around this by manually setting the MTU inside the namespace to 1274 to match the VPN.

Comment 8 Jonathan Lebon 2017-11-27 22:35:07 UTC
Any updates on this? This is a constant source of frustration and gotchas.

Comment 9 Daniel Walsh 2017-12-11 22:21:25 UTC
Eric, Mrunal and Dan, any ideas on this? Is this something we could fix with CRI-O?  Something fixable in CNI?

Comment 10 Colin Walters 2017-12-12 20:40:13 UTC
One thing I wonder is if we should try delegating at least some of the bridge device creation to NetworkManager, which is in somewhat of a better position to understand the host routing state.

Comment 11 Fedora End Of Life 2018-05-03 08:34:47 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 12 Fedora End Of Life 2018-05-29 11:59:32 UTC
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.