Bug 1460760
| Summary: | Virtio-net interface MTU overwritten to 1500 bytes | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Maxime Coquelin <maxime.coquelin> | ||||
| Component: | NetworkManager | Assignee: | Thomas Haller <thaller> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Desktop QE <desktop-qa-list> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 7.4 | CC: | aconole, ailan, atragler, bgalvani, fbaudin, fgiudici, laine, lmiksik, lrintel, maxime.coquelin, rbalakri, rkhan, sukulkar, thaller, vbenes, yalzhang | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | NetworkManager-1.8.0-9.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-08-01 09:30:33 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Could you enable debug logging (level=TRACE) in /etc/NetworkManager/NetworkManager.conf and attach a logfile that shows the issue? Thanks (In reply to Thomas Haller from comment #2) > Could you enable debug logging (level=TRACE) in > /etc/NetworkManager/NetworkManager.conf and attach a logfile that shows the > issue? More how-to here: https://cgit.freedesktop.org/NetworkManager/NetworkManager/plain/contrib/fedora/rpm/NetworkManager.conf?id=master A restart of NetworkManager after changing the log-level would be great. Created attachment 1287079 [details]
NetworkManager logs
Please find attached the requested log.
The affected interface is eth0.
I think the interesting bits are:
<debug> [1497285580.6423] dhcp4 (eth0): running: /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-e88fb32c-8f41-4392-8551-7ee44124efae-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
<info> [1497285580.6477] dhcp4 (eth0): dhclient started with pid 777
<debug> [1497285580.6479] device[0x55cc41885580] (eth0): add_pending_action (2): 'dhcp4'
<trace> [1497285580.6479] device[0x55cc41885580] (eth0): ip6-state: set to 2 (conf)
<trace> [1497285580.6479] device[0x55cc41885580] (eth0): mtu: device-mtu: 1500, ipv6-mtu: 1500, ifindex: 2
<debug> [1497285580.6480] platform-linux: sysctl: reading '/proc/sys/net/ipv6/conf/eth0/mtu': '9000'
<debug> [1497285580.6480] platform: link: setting 'eth0' (2) mtu 1500
<debug> [1497285580.6480] platform-linux: link: change 2: mtu: 1500
<debug> [1497285580.6481] platform-linux: do-request-link: 2
Note that I also dumped the traffic on host side, and can confirm the
DHCP server does not provide any MTU value in its reply.
To be more accurate, the client has "interface MTU value" set in its parameter request list,
but the server does not reply with this parameter.
When NetworkManager starts, it finds a connection ifcfg-eth0. The connection is: - autoconnecty=yes (ONBOOT) - doesn't specify a MTU size. Also, the eth0 device is not explicitly configured as unmanaged to NetworkManager. You could do so via multiple ways, for example by setting NM_CONTROLLED=no in the ifcfg file. When NM starts, all conditions are met and NM starts auto activating the connection. This involves setting the MTU. NetworkManager is doing as it is told. I am not sure who is responsible for creating the unsuitable ifcfg-eth0 file, or which component should do something differently. Reassigning to libvirt for investigation. I take this bug back. Sorry for the noise. In the past, in absence of explicit MTU configuration, NM would not reconfigure it. That was changed. I think this change in behavior might be problematic (in it's own right), and it also seems wrong to me now. Restored previous behavior: master: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=4ca3002b86948847711cd5b1937008baef3c30da nm-1-8: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=588841f2e086774420a7ff4452d87e45ffae578a Now, if neither the configuration nor DHCP (PPP, etc) specify an MTU, it is not modified during connection activation. That should avoid the present issue. I think the change in behavior might cause problems for rhel-7.4. Requesting blocker. For QA: It's easy to reproduce. On current rhel-7.4-beta, when activating an device NM will always set the MTU. It does so even if the connection doesn't specify the MTU nor an MTU is provided by DHCP. In that case, NM would configure the MTU as 1500 (for ethernet, actually not for infiniband). On rhel-7.3, it wouldn't. Now, the previous behavior is restored and NM does not touch the MTU unless explicitly configured. What however improved in rhel-7.4 compared to rhel-7.3, is that when deactivating a connection, the previously set MTU gets restored. In 7.3, NM would leave the MTU as it was when deactivating the device. You've already determined the source of the problem, but just for future info - in this case the ifcfg file is in the guest OS, so isn't created by libvirt, but by anaconda (or possible later modification by NM or manual editing by the user). Actually libvirt only modifies ifcfg files in the rare case that a user modifies host interface configuration via the libvirt API; this is fortunately not a very common practice. In the scenario described here, libvirt's only involvement is to 1) set the MTU of the tap (or macvtap) device that's created on the host, and 2) add the same MTU to the qemu commandline so that qemu can add it into the emulated guest NIC's PCI config data. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:2299 |
Description of problem: The new Virtio MTU feature enables the host to pass initial MTU value to the guest's Virtio-net device at init time. The problem is that NetworkManager overwrites this value to 1500 bytes at init time for the Virtio-net devices created at guest installation time (even if DHCP query failed). Version-Release number of selected component (if applicable): libvirt-3.2.0-9.el7.x86_64 qemu-kvm-rhev-2.9.0-8.el7.x86_64 Guest kernel: 3.10.0-679.el7.x86_64 NetworkManager-1.8.0-6.el7.x86_64 How reproducible: Create a new guest, change the MTU value for the virtio-net device in the guest's libvirt XML. Steps to Reproduce: 1. Create a new RHEL7.4 guest: host# virt-install -n rhel7.4_tmp --memory 3072 --os-type=linux --os-variant=rhel7 \ --disk /home/virt/rhel74_tmp.qcow2,size=20 \ --graphics vnc,port=5900,listen=0.0.0.0 \ --location http://download-node-02.eng.bos.redhat.com/rel-eng/RHEL-7.4-20170608.1/compose/Server/x86_64/os/ 2. Edit guest's XML to add MTU value: host# virsh edit rhel7.4_tmp Add MTU value as in below node: <interface type='network'> <mac address='52:54:00:58:d7:bd'/> <source network='default'/> <model type='virtio'/> <mtu size='9000'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> 3. Boot guest, and check virtio-net's interface MTU value guest# ip addr Actual results: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 52:54:00:58:d7:bd brd ff:ff:ff:ff:ff:ff Expected results: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 52:54:00:58:d7:bd brd ff:ff:ff:ff:ff:ff Additional info: From testings, possible workarounds identified to have the MTU value provided by the host: - Disable NetworkManager service and reboot - Change MAC address of the virtio-net device in the Guest's XML - Exclude the interface from NetworkManager control: cat /etc/sysconfig/network-scripts/ifcfg-eth0 # Generated by dracut initrd NAME="eth0" HWADDR="52:54:00:58:d7:bd" ONBOOT=yes NETBOOT=yes UUID="e88fb32c-8f41-4392-8551-7ee44124efae" IPV6INIT=yes BOOTPROTO=dhcp TYPE=Ethernet NM_CONTROLLED=no