Description of problem:
The new Virtio MTU feature enables the host to pass initial MTU value to the
guest's Virtio-net device at init time.
The problem is that NetworkManager overwrites this value to 1500 bytes at init
time for the Virtio-net devices created at guest installation time (even if
DHCP query failed).
Version-Release number of selected component (if applicable):
Guest kernel: 3.10.0-679.el7.x86_64
Create a new guest, change the MTU value for the virtio-net device in the guest's libvirt XML.
Steps to Reproduce:
1. Create a new RHEL7.4 guest:
host# virt-install -n rhel7.4_tmp --memory 3072 --os-type=linux --os-variant=rhel7 \
--disk /home/virt/rhel74_tmp.qcow2,size=20 \
--graphics vnc,port=5900,listen=0.0.0.0 \
2. Edit guest's XML to add MTU value:
host# virsh edit rhel7.4_tmp
Add MTU value as in below node:
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
3. Boot guest, and check virtio-net's interface MTU value
guest# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 52:54:00:58:d7:bd brd ff:ff:ff:ff:ff:ff
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
link/ether 52:54:00:58:d7:bd brd ff:ff:ff:ff:ff:ff
From testings, possible workarounds identified to have the MTU value provided by the host:
- Disable NetworkManager service and reboot
- Change MAC address of the virtio-net device in the Guest's XML
- Exclude the interface from NetworkManager control:
# Generated by dracut initrd
Could you enable debug logging (level=TRACE) in /etc/NetworkManager/NetworkManager.conf and attach a logfile that shows the issue?
(In reply to Thomas Haller from comment #2)
> Could you enable debug logging (level=TRACE) in
> /etc/NetworkManager/NetworkManager.conf and attach a logfile that shows the
More how-to here: https://cgit.freedesktop.org/NetworkManager/NetworkManager/plain/contrib/fedora/rpm/NetworkManager.conf?id=master
A restart of NetworkManager after changing the log-level would be great.
Created attachment 1287079 [details]
Please find attached the requested log.
The affected interface is eth0.
I think the interesting bits are:
<debug> [1497285580.6423] dhcp4 (eth0): running: /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-e88fb32c-8f41-4392-8551-7ee44124efae-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
<info> [1497285580.6477] dhcp4 (eth0): dhclient started with pid 777
<debug> [1497285580.6479] device[0x55cc41885580] (eth0): add_pending_action (2): 'dhcp4'
<trace> [1497285580.6479] device[0x55cc41885580] (eth0): ip6-state: set to 2 (conf)
<trace> [1497285580.6479] device[0x55cc41885580] (eth0): mtu: device-mtu: 1500, ipv6-mtu: 1500, ifindex: 2
<debug> [1497285580.6480] platform-linux: sysctl: reading '/proc/sys/net/ipv6/conf/eth0/mtu': '9000'
<debug> [1497285580.6480] platform: link: setting 'eth0' (2) mtu 1500
<debug> [1497285580.6480] platform-linux: link: change 2: mtu: 1500
<debug> [1497285580.6481] platform-linux: do-request-link: 2
Note that I also dumped the traffic on host side, and can confirm the
DHCP server does not provide any MTU value in its reply.
To be more accurate, the client has "interface MTU value" set in its parameter request list,
but the server does not reply with this parameter.
When NetworkManager starts, it finds a connection ifcfg-eth0.
The connection is:
- autoconnecty=yes (ONBOOT)
- doesn't specify a MTU size.
Also, the eth0 device is not explicitly configured as unmanaged to NetworkManager. You could do so via multiple ways, for example by setting NM_CONTROLLED=no in the ifcfg file.
When NM starts, all conditions are met and NM starts auto activating the connection. This involves setting the MTU.
NetworkManager is doing as it is told.
I am not sure who is responsible for creating the unsuitable ifcfg-eth0 file, or which component should do something differently. Reassigning to libvirt for investigation.
I take this bug back. Sorry for the noise.
In the past, in absence of explicit MTU configuration, NM would not reconfigure it. That was changed.
I think this change in behavior might be problematic (in it's own right), and it also seems wrong to me now.
Restored previous behavior:
Now, if neither the configuration nor DHCP (PPP, etc) specify an MTU, it is not modified during connection activation. That should avoid the present issue.
I think the change in behavior might cause problems for rhel-7.4. Requesting blocker.
It's easy to reproduce.
On current rhel-7.4-beta, when activating an device NM will always set the MTU. It does so even if the connection doesn't specify the MTU nor an MTU is provided by DHCP.
In that case, NM would configure the MTU as 1500 (for ethernet, actually not for infiniband).
On rhel-7.3, it wouldn't.
Now, the previous behavior is restored and NM does not touch the MTU unless explicitly configured.
What however improved in rhel-7.4 compared to rhel-7.3, is that when deactivating a connection, the previously set MTU gets restored. In 7.3, NM would leave the MTU as it was when deactivating the device.
You've already determined the source of the problem, but just for future info - in this case the ifcfg file is in the guest OS, so isn't created by libvirt, but by anaconda (or possible later modification by NM or manual editing by the user). Actually libvirt only modifies ifcfg files in the rare case that a user modifies host interface configuration via the libvirt API; this is fortunately not a very common practice.
In the scenario described here, libvirt's only involvement is to 1) set the MTU of the tap (or macvtap) device that's created on the host, and 2) add the same MTU to the qemu commandline so that qemu can add it into the emulated guest NIC's PCI config data.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.