Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: In https://github.com/moby/moby/pull/13953, docker was changed to inherit the MTU from the default route. Later in https://github.com/moby/moby/pull/18108, docker was changed back to *not* inherit from the default route and just let the kernel figure out the optimal connection MTU using path MTU discovery. However, I'm observing here that this is still not working and that one essentially has to use the --mtu option. E.g. in an OpenStack VM running the latest F26 AH: # ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000 <snip> # docker run --rm registry.fedoraproject.org/fedora:26 sh -c 'dnf install -y ip route && ip addr' <snip> 14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default <snip> This mismatch is fine, and IIUC in https://github.com/moby/moby/pull/18108, the kernel should be able to figure it out. But that doesn't seem to happen and as a result, we get severe network performance issues. So there's something wrong along the networking stack: either it's something in the kernel (e.g. a regression from https://bugzilla.redhat.com/show_bug.cgi?id=1441310 perhaps?), or there's something in docker preventing the kernel from doing its job, or there something wonky in OpenStack in the way it handles packages larger than $MTU. Version-Release number of selected component (if applicable): # rpm-ostree status State: idle Deployments: * fedora-atomic:fedora/26/x86_64/updates/atomic-host Version: 26.91 (2017-07-23 03:12:08) Commit: 0715ce81064c30d34ed52ef811a3ad5e5d6a34da980bf35b19312489b32d9b83 GPGSignature: 1 signature Signature made Sun Jul 23 03:13:09 2017 using RSA key ID 812A6B4B64DAB85D Good signature from "Fedora 26 Primary <fedora-26-primary>" # rpm -q docker kernel docker-1.13.1-19.git27e468e.fc26.x86_64 kernel-4.11.11-300.fc26.x86_64 How reproducible: Always Steps to Reproduce: 1. Boot F26 AH in OpenStack 2. Start docker 3. Start a container 4. Try to curl something from an internal site, e.g.: curl -L http://download.devel.redhat.com/pub/fedora/linux/updates/26/x86_64/repodata/repomd.xml Actual results: curl invocation hangs Expected results: curl invocation works Additional info: As expected, changing options to e.g. --mtu 1400 allows the curl request to go through (or alternatively, creating the container with --net=host).
This looks useful: https://blog.cloudflare.com/path-mtu-discovery-in-practice/
It looks like in this: https://github.com/projectcalico/calicoctl/issues/488 They discussed ensuring the MTU isn't *larger* than the host routes, which makes sense to me. Also found: https://github.com/openshift/openshift-ansible/pull/544/commits/c28aea2f19b6aea729f27b636e5318ac775addb6 which looks to me like it hardcodes a default of 1450?
This could possibly go back as far as docker 1.10. We've been hitting this for a long time. See the following, which I believe are all related. https://bugzilla.redhat.com/show_bug.cgi?id=1333961 https://bugzilla.redhat.com/show_bug.cgi?id=1357116 https://projects.engineering.redhat.com/browse/CENTRALCI-1749 http://post-office.corp.redhat.com/archives/ci-infra-list/2017-March/msg00036.html
*** Bug 1474793 has been marked as a duplicate of this bug. ***
I see this constantly on Fedora 26 with docker or with firejail. In both cases I see the issue when I have a network namespace (from docker or firejail) connected to a bridged network interface and have an openconnect VPN session open. In these cases all MTUs are 1500, except vpn0 (from openconnect) with an MTU of 1274. Traffic from outside a namespace is fine, going through the vpn or not. Traffic from inside the name space that doesn't go oever the VPN is fine. But when it goes over the VPN, tcpdump shows large response packets are getting sent by remote servers, then dropped by vpn0. I was able to work around this by manually setting the MTU inside the namespace to 1274 to match the VPN.
Any updates on this? This is a constant source of frustration and gotchas.
Eric, Mrunal and Dan, any ideas on this? Is this something we could fix with CRI-O? Something fixable in CNI?
One thing I wonder is if we should try delegating at least some of the bridge device creation to NetworkManager, which is in somewhat of a better position to understand the host routing state.
This message is a reminder that Fedora 26 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 26. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '26'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 26 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.