Bug 1460760 - Virtio-net interface MTU overwritten to 1500 bytes
Summary: Virtio-net interface MTU overwritten to 1500 bytes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Thomas Haller
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-12 16:06 UTC by Maxime Coquelin
Modified: 2017-08-01 09:30 UTC (History)
16 users (show)

Fixed In Version: NetworkManager-1.8.0-9.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 09:30:33 UTC


Attachments (Terms of Use)
NetworkManager logs (147.27 KB, text/x-vhdl)
2017-06-12 17:27 UTC, Maxime Coquelin
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:2299 normal SHIPPED_LIVE Moderate: NetworkManager and libnl3 security, bug fix and enhancement update 2017-08-01 12:40:28 UTC
Red Hat Bugzilla 1412234 None CLOSED extend virtio-net to expose host MTU to guest 2019-08-23 21:55:29 UTC
Red Hat Bugzilla 1452756 None None None 2019-08-23 21:55:29 UTC

Internal Links: 1412234 1452756

Description Maxime Coquelin 2017-06-12 16:06:29 UTC
Description of problem:

The new Virtio MTU feature enables the host to pass initial MTU value to the 
guest's Virtio-net device at init time.

The problem is that NetworkManager overwrites this value to 1500 bytes at init 
time for the Virtio-net devices created at guest installation time (even if
DHCP query failed).

Version-Release number of selected component (if applicable):

libvirt-3.2.0-9.el7.x86_64
qemu-kvm-rhev-2.9.0-8.el7.x86_64
Guest kernel: 3.10.0-679.el7.x86_64
NetworkManager-1.8.0-6.el7.x86_64

How reproducible:
Create a new guest, change the MTU value for the virtio-net device in the guest's libvirt XML.

Steps to Reproduce:
1. Create a new RHEL7.4 guest:

host# virt-install -n rhel7.4_tmp --memory 3072 --os-type=linux --os-variant=rhel7 \
                   --disk /home/virt/rhel74_tmp.qcow2,size=20 \
                   --graphics vnc,port=5900,listen=0.0.0.0 \
                   --location http://download-node-02.eng.bos.redhat.com/rel-eng/RHEL-7.4-20170608.1/compose/Server/x86_64/os/

2. Edit guest's XML to add MTU value:

host# virsh edit rhel7.4_tmp
Add MTU value as in below node:
    <interface type='network'>
      <mac address='52:54:00:58:d7:bd'/>
      <source network='default'/>
      <model type='virtio'/>
      <mtu size='9000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

3. Boot guest, and check virtio-net's interface MTU value

guest# ip addr

Actual results:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:58:d7:bd brd ff:ff:ff:ff:ff:ff

Expected results:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:58:d7:bd brd ff:ff:ff:ff:ff:ff

Additional info:


From testings, possible workarounds identified to have the MTU value provided by the host:
- Disable NetworkManager service and reboot
- Change MAC address of the virtio-net device in the Guest's XML
- Exclude the interface from NetworkManager control:

cat /etc/sysconfig/network-scripts/ifcfg-eth0 
# Generated by dracut initrd
NAME="eth0"
HWADDR="52:54:00:58:d7:bd"
ONBOOT=yes
NETBOOT=yes
UUID="e88fb32c-8f41-4392-8551-7ee44124efae"
IPV6INIT=yes
BOOTPROTO=dhcp
TYPE=Ethernet
NM_CONTROLLED=no

Comment 2 Thomas Haller 2017-06-12 16:35:38 UTC
Could you enable debug logging (level=TRACE) in /etc/NetworkManager/NetworkManager.conf and attach a logfile that shows the issue?

Thanks

Comment 3 Thomas Haller 2017-06-12 16:36:53 UTC
(In reply to Thomas Haller from comment #2)
> Could you enable debug logging (level=TRACE) in
> /etc/NetworkManager/NetworkManager.conf and attach a logfile that shows the
> issue?

More how-to here: https://cgit.freedesktop.org/NetworkManager/NetworkManager/plain/contrib/fedora/rpm/NetworkManager.conf?id=master
A restart of NetworkManager after changing the log-level would be great.

Comment 4 Maxime Coquelin 2017-06-12 17:27:03 UTC
Created attachment 1287079 [details]
NetworkManager logs

Please find attached the requested log.
The affected interface is eth0.

I think the interesting bits are:
<debug> [1497285580.6423] dhcp4 (eth0): running: /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-e88fb32c-8f41-4392-8551-7ee44124efae-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
<info>  [1497285580.6477] dhcp4 (eth0): dhclient started with pid 777
<debug> [1497285580.6479] device[0x55cc41885580] (eth0): add_pending_action (2): 'dhcp4'
<trace> [1497285580.6479] device[0x55cc41885580] (eth0): ip6-state: set to 2 (conf)
<trace> [1497285580.6479] device[0x55cc41885580] (eth0): mtu: device-mtu: 1500, ipv6-mtu: 1500, ifindex: 2
<debug> [1497285580.6480] platform-linux: sysctl: reading '/proc/sys/net/ipv6/conf/eth0/mtu': '9000'
<debug> [1497285580.6480] platform: link: setting 'eth0' (2) mtu 1500
<debug> [1497285580.6480] platform-linux: link: change 2: mtu: 1500
<debug> [1497285580.6481] platform-linux: do-request-link: 2

Note that I also dumped the traffic on host side, and can confirm the
DHCP server does not provide any MTU value in its reply.
To be more accurate, the client has "interface MTU value" set in its parameter request list,
but the server does not reply with this parameter.

Comment 5 Thomas Haller 2017-06-13 09:08:59 UTC
When NetworkManager starts, it finds a connection ifcfg-eth0.
The connection is:
  - autoconnecty=yes (ONBOOT)
  - doesn't specify a MTU size.

Also, the eth0 device is not explicitly configured as unmanaged to NetworkManager. You could do so via multiple ways, for example by setting NM_CONTROLLED=no in the ifcfg file.

When NM starts, all conditions are met and NM starts auto activating the connection. This involves setting the MTU.


NetworkManager is doing as it is told.



I am not sure who is responsible for creating the unsuitable ifcfg-eth0 file, or which component should do something differently. Reassigning to libvirt for investigation.

Comment 6 Thomas Haller 2017-06-13 13:33:00 UTC
I take this bug back. Sorry for the noise.



In the past, in absence of explicit MTU configuration, NM would not reconfigure it. That was changed.

I think this change in behavior might be problematic (in it's own right), and it also seems wrong to me now.



Restored previous behavior:

master: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=4ca3002b86948847711cd5b1937008baef3c30da

nm-1-8: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=588841f2e086774420a7ff4452d87e45ffae578a



Now, if neither the configuration nor DHCP (PPP, etc) specify an MTU, it is not modified during connection activation. That should avoid the present issue.




I think the change in behavior might cause problems for rhel-7.4. Requesting blocker.

Comment 7 Thomas Haller 2017-06-13 13:46:57 UTC
For QA:

It's easy to reproduce.

On current rhel-7.4-beta, when activating an device NM will always set the MTU. It does so even if the connection doesn't specify the MTU nor an MTU is provided by DHCP.
In that case, NM would configure the MTU as 1500 (for ethernet, actually not for infiniband).

On rhel-7.3, it wouldn't.

Now, the previous behavior is restored and NM does not touch the MTU unless explicitly configured.



What however improved in rhel-7.4 compared to rhel-7.3, is that when deactivating a connection, the previously set MTU gets restored. In 7.3, NM would leave the MTU as it was when deactivating the device.

Comment 8 Laine Stump 2017-06-13 19:03:11 UTC
You've already determined the source of the problem, but just for future info - in this case the ifcfg file is in the guest OS, so isn't created by libvirt, but  by anaconda (or possible later modification by NM or manual editing by the user). Actually libvirt only modifies ifcfg files in the rare case that a user modifies host interface configuration via the libvirt API; this is fortunately not a very common practice.

In the scenario described here, libvirt's only involvement is to 1) set the MTU of the tap (or macvtap) device that's created on the host, and 2) add the same MTU to the qemu commandline so that qemu can add it into the emulated guest NIC's PCI config data.

Comment 11 errata-xmlrpc 2017-08-01 09:30:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2299


Note You need to log in before you can comment on or make changes to this bug.