As a "side feature" to this, we should provide a config knob to set the MTU of the bridge in a libvirt virtual network. This seems slightly problematic though since, as someone pointed out in #virt on oftc, the kernel won't allow setting an MTU over 1500 on a bridge device that doesn't have a physical ethernet attached.

Vlad - do you know why that is? (and more importantly, how to fix it?)

The bridge will always configure to the lowest set mtu of all the devices added.  This means adding a new virtual interface to the bridge (or changing the mtu of an existing bridge element) will reconfigure the bridge MTU.

Is there a reason to care on an empty bridge?

As far as the fix goes, we can change the code for br_min_mtu to return the max-value.

Does this help as far as information?

Okay, I looked at this briefly just before leaving for a 12 day vacation, and now realize that I misremembered a bit, and also that I hadn't completed my investigation - my recollection was that the new tap devices being added would match the MTU of the bridge (I was thinking of it this way because libvirt always checks the MTU of the bridge prior to attaching a tap device, then sets the tap device MTU to match the bridge), but as you point out it's the opposite.

Beyond that, I just now tried the thing that I had *intended* to try before I left town (but didn't get around to, and then forgot) - if I 1) create a bridge device, then 2) create a tap device and 3) set the tap MTU to, e.g. 9000, and *then* 4) attach the tap to the bridge, the bridge now has a 9000 MTU. Since libvirt already creates a "dummy" tap device to attach to every bridge it creates (for multiple bookkeeping reasons), all we need to do is set the MTU of that tap device to the MTU we want for the network, and the bridge, as well as all new tap devices will inherit that MTU. So no changes are needed from anyone outside libvirt.

Thanks for pushing me to actually experiment, and sorry for the digression.

Is there a reason why this BZ, and its parent qemu BZ are marked private? I don't see any sensitive info anywhere. Marking them private means they can't be referred to in any public commit log or discussion, which leads to information disconnects in the future.

(In reply to Laine Stump from comment #4)
> Is there a reason why this BZ, and its parent qemu BZ are marked private? I
> don't see any sensitive info anywhere. Marking them private means they can't
> be referred to in any public commit log or discussion, which leads to
> information disconnects in the future.

You are right, changed it.

I've just pushed the patches upstream:

commit 572eda12ad7336031fbbdc8c130c8674ba7764e8
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jan 23 14:33:20 2017 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jan 26 10:00:01 2017 +0100

    qemu: Implement mtu on interface
    
    Not only we should set the MTU on the host end of the device but
    also let qemu know what MTU did we set.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit b020cf73fed761067c3b6196d993545f072c896f
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jan 23 14:32:13 2017 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Jan 26 09:59:56 2017 +0100

    domain_conf: Introduce <mtu/> to <interface/>
    
    So far we allow to set MTU for libvirt networks. However, not all
    domain interfaces have to be plugged into a libvirt network and
    even if they are, they might want to have a different MTU (e.g.
    for testing purposes).
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit eebec1697e88080f5a1271d104083430e34c457f
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jan 23 12:58:23 2017 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Wed Jan 25 09:18:49 2017 +0100

    virDomainNetDefParseXML: s/ret/rv/
    
    We use @ret to hold the actual return value of the function we
    are currently in. To hold a return value of a function called we
    use different variables: @rv, @rc, etc. Honour this naming
    scheme in virDomainNetDefParseXML too.
    
    Signed-off-by: Michal Privoznik <mprivozn>


v3.0.0-57-g572eda12a

Michal - Amnon asked the other day if we can learn the correct MTU for a vhost-user interface "from the switch", as it would be much simpler to configure if the user only needed to set the MTU for a switch, and then libvirt would query the switch MTU and set the appropriate host_mtu for all guest vhost-user interfaces using that switch.

I've already added on to my patches (but not yet posted - still cleaning up and testing) to set network MTU so that they will propogate the switch's MTU back to the qemu commandline in the case of Linux host bridge and OVS bridge connections that use tap devices (so we can just query the MTU of the device with virNetDevGetMTU()) but vhost-user connects to a Unix socket, and I know *nothing* about what's beyond that, or how to get to it. Do you know how to learn the MTU of the network in that case?

(In reply to Laine Stump from comment #7)
> Michal - Amnon asked the other day if we can learn the correct MTU for a
> vhost-user interface "from the switch", as it would be much simpler to
> configure if the user only needed to set the MTU for a switch, and then
> libvirt would query the switch MTU and set the appropriate host_mtu for all
> guest vhost-user interfaces using that switch.
> 
> I've already added on to my patches (but not yet posted - still cleaning up
> and testing) to set network MTU so that they will propogate the switch's MTU
> back to the qemu commandline in the case of Linux host bridge and OVS bridge
> connections that use tap devices (so we can just query the MTU of the device
> with virNetDevGetMTU()) but vhost-user connects to a Unix socket, and I know
> *nothing* about what's beyond that, or how to get to it. Do you know how to
> learn the MTU of the network in that case?

Sure:

# ovs-vsctl get Interface ovsbr0 mtu
1500

However, why do we need it on cmd line for classic Linux bridge + tap device case? Libvirt automatically sets tap device's mtu to match the one that the bridge has (unless the tap device has different MTU set - in which case we also put it onto qemu's cmd line).

(In reply to Michal Privoznik from comment #8)

> 
> However, why do we need it on cmd line for classic Linux bridge + tap device
> case? Libvirt automatically sets tap device's mtu to match the one that the
> bridge has (unless the tap device has different MTU set - in which case we
> also put it onto qemu's cmd line).

I don't see that anywhere in the code. It's true that we set the tap device MTU to match the bridge, but (without the patch that I have locally) this updated MTU isn't propagated into a host_mtu option on the commandline - host_mtu is only added if <mtu size...> is set in the interface config. (Also, for OVS bridges, libvirt is already getting the MTU of the bridge using a normal ioctl, just as it does for linux host bridges. Does this not work properly? (I guess I need to install OVS on my test machine to see).

As far as using ovs-vsctl to get the MTU of the OVS bridge for vhost-user interfaces - how do we know which bridge's MTU to get? All we know in the case of vhost-user is the name of the unix socket.

The patches below were pushed upstream and will be in libvirt 3.1.0. They take care of the issue I mentioned in Comment 9, as long as the network uses tap devices to connect to a Linux host bridge or OVS bridge.

For vhost-user, the discussion on the ML indicates that we'll just have to rely on Nova (or whatever higher level management application knows the details of what's at the other end of the Unix socket used by vhost-users) adding an <mtu size='blah'/> to the domain interface config.


commit dd8ac030fbd28bba81c24cdf1311e47a350a7683
Author: Laine Stump <laine>
Date:   Sun Jan 22 20:41:03 2017 -0500

    util: add MTU arg to virNetDevTapCreateInBridgePort()
    
commit 68a42bf6f701515df472f0dd039a1d7429ea62a8
Author: Laine Stump <laine>
Date:   Sun Jan 22 21:23:48 2017 -0500

    conf: support configuring mtu size in a virtual network
    
commit c0f706865e4ecac0ddaafdb4b9fc2db8d9612481
Author: Laine Stump <laine>
Date:   Sun Jan 22 21:33:07 2017 -0500

    network: honor mtu setting when creating network
    


commit 2841e6756d5807a4119e004bc5fb8e7d70806458
Author: Laine Stump <laine>
Date:   Fri Feb 3 11:55:20 2017 -0500

    qemu: propagate bridge MTU into qemu "host_mtu" option

(In reply to Laine Stump from comment #10)
> The patches below were pushed upstream and will be in libvirt 3.1.0. They
> take care of the issue I mentioned in Comment 9, as long as the network uses
> tap devices to connect to a Linux host bridge or OVS bridge.
> 
> For vhost-user, the discussion on the ML indicates that we'll just have to
> rely on Nova (or whatever higher level management application knows the
> details of what's at the other end of the Unix socket used by vhost-users)
> adding an <mtu size='blah'/> to the domain interface config.
> 

Why is OVS bridge different from OVS-DPDK (vhost-user)? Isn't there 
a common DB to query?

From libbirt's point of view a vhost-user interface doesn't connect "to OVS", it connects to a Unix socket. It has no idea that it is OVS at the other end. And from a discussion thread on libvir-list, I think consensus is that we don't want libvirt to know anything about what's at the other end of the socket. e.g.:

  https://www.redhat.com/archives/libvir-list/2017-February/msg00066.html

Also, a tap-based connection doesn't do any OVS-specific query to determine the MTU of the bridge - it just does a normal ioctl(SIOCGIFMTU), just as it does for a Linux host bridge.

Test on below packages:

libvirt-3.2.0-9.el7.x86_64
qemu-kvm-rhev-2.9.0-8.el7.x86_64

1. hotplug an interface device with mtu setting  ---> FAIL, the tap device will not get the mtu setting.

# cat inter.xml
    <interface type='network'>
      <source network='default' bridge='virbr0'/>
      <model type='virtio'/>
      <mtu size='9000'/>
    </interface>

# virsh attach-device rhel7 inter.xml
Device attached successfully

# virsh dumpxml rhel7 | grep /interface -B8
    <interface type='network'>
      <mac address='52:54:00:0b:42:70'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <mtu size='9000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

# ip link show vnet0
182: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN mode DEFAULT qlen 1000
    link/ether fe:54:00:0b:42:70 brd ff:ff:ff:ff:ff:ff

# virsh qemu-monitor-command --hmp rhel7 info network
net0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:0b:42:70
 \ hostnet0: index=0,type=tap,fd=95

2. update the mtu setting by update-device ---> FAIL, refer to bug 1447618.

(In reply to yalzhang from comment #15)
> Test on below packages:
> 
> libvirt-3.2.0-9.el7.x86_64
> qemu-kvm-rhev-2.9.0-8.el7.x86_64
> 
> 1. hotplug an interface device with mtu setting  ---> FAIL, the tap device
> will not get the mtu setting.

Ah, nice catch. Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2017-June/msg00438.html

Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2017-June/msg00213.html

One more scenario need your confirmation, it is something like Bug 1311928
Do you think it is acceptable?

1. the "hostnet" is a hostdev network

# virsh dumpxml V2 | grep /interface -B7 
    <interface type='network'>
      <mac address='52:54:00:44:fd:d8'/>
      <source network='hostnet'/>
      <model type='rtl8139'/>
      <mtu size='9000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
2.
# virsh start V2
Domain V2 started

# virsh dumpxml V2 | grep /interface -B11
   <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:44:fd:d8'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x86' slot='0x10' function='0x0'/>
      </source>
      <model type='rtl8139'/>
      <mtu size='9000'/>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

3. when you add mtu setting in <interface type = "hostdev">, it will fail, this is expected.

# virsh edit V
    <interface type='hostdev'>
      <mac address='52:54:00:99:13:e2'/>
<mtu size='9000'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x86' slot='0x10' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


error: unsupported configuration: setting MTU on interface type hostdev is not supported yet
Failed. Try again? [y,n,i,f,?]:

(In reply to yalzhang from comment #20)
> One more scenario need your confirmation, it is something like Bug 1311928
> Do you think it is acceptable?

I think it's acceptable for the feature. I mean, the feature is there and working.

> 
> 1. the "hostnet" is a hostdev network
> 
> # virsh dumpxml V2 | grep /interface -B7 
>     <interface type='network'>
>       <mac address='52:54:00:44:fd:d8'/>
>       <source network='hostnet'/>
>       <model type='rtl8139'/>
>       <mtu size='9000'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>     </interface>
> 2.
> # virsh start V2
> Domain V2 started
> 
> # virsh dumpxml V2 | grep /interface -B11
>    <interface type='hostdev' managed='yes'>
>       <mac address='52:54:00:44:fd:d8'/>
>       <driver name='vfio'/>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x86' slot='0x10'
> function='0x0'/>
>       </source>
>       <model type='rtl8139'/>
>       <mtu size='9000'/>
>       <alias name='hostdev0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>     </interface>
> 
> 3. when you add mtu setting in <interface type = "hostdev">, it will fail,
> this is expected.
> 
> # virsh edit V
>     <interface type='hostdev'>
>       <mac address='52:54:00:99:13:e2'/>
> <mtu size='9000'/>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x86' slot='0x10'
> function='0x1'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
> 
> 
> error: unsupported configuration: setting MTU on interface type hostdev is
> not supported yet
> Failed. Try again? [y,n,i,f,?]:

And for this I think we can have a separate bug. While being not specifically nice, it is a harmless corner case.

There's really not anything we can do about the inconsistency in error reporting at domain definition time - when the interface is type='network', you don't know until you actually instantiate the interface whether it will be a standard tap device, or macvtap or hostdev. We have the same issue with setting the model name with <interface type='network'> - libvirt automatically adds "<model type='rtl8139'/>", which is nonsensical in the case of a hostdev interface, but since we can't know what the interface type *is going to be*, we can't report an error at that time (and in this case, since it is a value automatically added by libvirt, we can't even report it as an error at a later time - that's something we *can* do for an erroneous MTU setting though).

As for the idea of doing something productive with the MTU setting in the XML for type='hostdev', I haven't tried it, but my guess is that any MTU setting made to a VF prior to detaching it from the hostside net driver will be obliterated during the process of rebinding to vfio-pci and then initializing the VF via the VF netdriver on the guest. (I suspect this because there is no place in the PF's VFINFO to store an MTU (as there is for MAC address and vlan tag), so unless it's stored on the hardware itself in a manner that makes it involatile in the face of a re-init, it will be lost.)

(In reply to Laine Stump from comment #23)

> As for the idea of doing something productive with the MTU setting in the
> XML for type='hostdev', I haven't tried it, but my guess is that any MTU
> setting made to a VF prior to detaching it from the hostside net driver will
> be obliterated during the process of rebinding to vfio-pci and then
> initializing the VF via the VF netdriver on the guest. (I suspect this
> because there is no place in the PF's VFINFO to store an MTU (as there is
> for MAC address and vlan tag), so unless it's stored on the hardware itself
> in a manner that makes it involatile in the face of a re-init, it will be
> lost.)

Exactly.  When the driver is loaded in the guest, the MTU will be set to default
ethernet MTU by default.

In fact, thinking about it a bit more, mtu value doesn't really make sense for any devices other then virtio, since only virtio has the capability to propagate
that value to the guest.

-vlad

(In reply to Vlad Yasevich from comment #24)

> In fact, thinking about it a bit more, mtu value doesn't really make sense
> for any devices other then virtio, since only virtio has the capability to
> propagate that value to the guest.

Almost. libvirt sets the MTU of the tap/macvtap device to the provided value. This part happens regardless of whether or not the emulated device model is virtio.

(In reply to Laine Stump from comment #25)
> (In reply to Vlad Yasevich from comment #24)
> 
> > In fact, thinking about it a bit more, mtu value doesn't really make sense
> > for any devices other then virtio, since only virtio has the capability to
> > propagate that value to the guest.
> 
> Almost. libvirt sets the MTU of the tap/macvtap device to the provided
> value. This part happens regardless of whether or not the emulated device
> model is virtio.

Ok.  I guess it might be a good idea for larger MTU values, but to take advantage
of this, the guest admin would have to manually set the mtu on the guest.

It may end up being pointless if another VM joins the bridge with a lower MTU value, but that's a generic problem with the way MTU was proposed.

Question:  For macvtap, does libvirt change the physical device mtu as well for macvtap?  If not, then setting macvtap mtu > physical device mtu will result in error from the kernel.

-vlad

(In reply to Vlad Yasevich from comment #26)
 
> Question:  For macvtap, does libvirt change the physical device mtu as well
> for macvtap?  If not, then setting macvtap mtu > physical device mtu will
> result in error from the kernel.
> 
> -vlad

set mtu on <interface type='direct'>  is not supported, 
and set mtu on <interface type='network'> with macvtap network, the guest can not start.

1. # virsh edit rhel7.4
   <interface type='direct'>
      <mac address='52:54:00:08:c9:9f'/>
      <source dev='enp0s25' mode='bridge'/>
      <model type='virtio'/>
        <mtu size='9000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

error: unsupported configuration: setting MTU on interface type direct is not supported yet
Failed. Try again? [y,n,i,f,?]:   ====> press 'i'

error: unsupported configuration: setting MTU on interface type direct is not supported yet
Failed. Try again? [y,n,f,?]: 


2. # virsh dumpxml rhel7.4 | grep /interface -B6
    <interface type='network'>
      <mac address='52:54:00:8d:47:6c'/>
      <source network='direct-macvtap'/>
      <model type='virtio'/>
      <mtu size='9000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </interface>
# virsh start rhel7.4
error: Failed to start domain rhel7.4
error: Cannot set interface MTU on 'macvtap0': Invalid argument

And when I set mtu on the guest for different network model type, I found the limitation is different. It may be something about the driver, I'm not sure. If there is anything unexpected?

for tap/tun on host, max mtu we can set is 65521
for rtl8139 on guest, max mtu can set is 4096
for e1000 on guest, max mtu can set is 16110
for e1000e on guest, max mtu can set is 9212

for virtio, guest will got mtu automatically, if modify mtu manually on guest, the value will be unacceptable if it is bigger than the setting in xml, this is expected.(with NetworkManager disabled)

if set <mtu size='8000'> in xml with <model type='virtio'/>
on guest:
# ifconfig eth0 | grep mtu
eth0: flags=4098<BROADCAST,MULTICAST>  mtu 8000

# ifconfig eth0 mtu 9000
SIOCSIFMTU: Invalid argument

if do not set mtu size for virtio in xml, the max mtu we can set in guest is 65535.

Test on below packages, the result is as expected, set this bug to be verified.

libvirt-3.2.0-10.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64

1. set mtu in interface ---> PASS

2. set mtu in nat network ---> PASS

3. set mtu both in network and interface ---> PASS

4. test boundary: negative value, character, blank space, big numbers ---> PASS

5. set mtu in other type network or interface ---> PASS
   
   1) set mtu in macvtap network or interface (comment #22);

   2) set mtu in hostdev network or interface (comment #20);

   3) set mtu in bridge type network or interface with an existing host bridge or ovs bridge;

6. cross migration ---> PASS (refer to Bug 1449346#c13)

7. migration ---> PASS

8. hotplug/unplug ---> PASS

9. update-device to update mtu ---> refer to bug 1447618, will be fixed rhel7.5


-----------Some detailed scenarios----------------------

Scenario 5. 3) set mtu in bridge type network or interface with an existing host bridge or ovs bridge;

1. prepare the host bridge and vm
# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.00247e054232	no		enp0s25

# ifconfig | grep mtu
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
enp0s25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500


# virsh dumpxml rhel7.4 | grep /interface -B6
    <interface type='bridge'>
      <mac address='52:54:00:d5:98:38'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <mtu size='9000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2. start the vm
# virsh start rhel7.4
Domain rhel7.4 started

3. 
# ifconfig | grep mtu
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
enp0s25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000

login guest and check the mtu is 9000

4. set mtu in bridge type network is not supported.
# cat br.xml
<network>
  <name>host-bridge</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
<mtu size='9000'/>
</network>

# virsh net-define br.xml
error: Failed to define network from br.xml
error: XML error: mtu size only allowed in open, route, nat, and isolated mode, not in bridge (network 'host-bridge')


Scenario 8: hotplug/unplug
 
# virsh start rhel7.4
Domain rhel7.4 started

 # cat interface.xml
<interface type='network'>
<source network='default'/>
<model type='virtio'/>
<mtu size='9000'/>
</interface>

# virsh attach-device rhel7.4 interface.xml;
Device attached successfully

# virsh dumpxml rhel7.4 | grep /interface -B8
    <interface type='network'>
      <mac address='52:54:00:a9:8a:d1'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <mtu size='9000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

# ifconfig vnet0 | grep mtu
vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000

login guest, check the mtu
# ifconfig eth0 | grep mtu
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846