Bug 1057321

Correct, the network-wide <bandwidth> element of a <network> is only applied to networks with a forward mode of 'route', 'nat', or none. In these cases libvirt creates its own bridge, and we know that all traffic to the rest of the world from the bridge will go via the bridge's own interface (e.g. "virbr0"), so we have a place to apply the bandwidth limits.

In the case of a <forward mode='bridge'/> network, the bridge was created by some other config on the system, and traffic from the guests to the rest of the network will likely *not* go via the bridge's own interface ('ovirtmgmt' in your case), so applying bandwidth controls there will do us no good. Rather, the most likely candidate for applying the network-wide bandwidth would be the physical adapter that is attached to the bridge (e.g. eth0), but libvirt has no information about that - it could learn it, but then would need to make a decision of whether or not that was the correct place - perhaps this bridge has two ethernet adapters attached, etc.

The real problem is that we are silently ignoring this unsupported config, rather than logging an error. I've posted a patch to remedy that upstream:

  https://www.redhat.com/archives/libvir-list/2014-January/msg01163.html

In the meantime, did you actually want to limit the entire network to those numbers? Or did you maybe just want to limit each guest interface to those numbers? If the latter, you can just add this to the network definition:

<portgroup name='default' default='yes'>
  <bandwidth>
    <inbound average='128' peak='128' burst='1024'/>
    <outbound average='128' peak='128' burst='1024'/>
  </bandwidth>
</portgroup>

When a guest interface is initialized, it will automatically apply the properties from the default portgroup, so those bandwidth properties will be applied to the single tap device that is connecting the guest's netdev.

Thanks for confirming this and for providing such a detailed explanation.

oVirt already has support for applying bandwidth limits to individual VM network interfaces.  We are now adding support for limiting the amount of traffic associated with a libvirt network that can flow over the physical host interface.  More context on the feature and how it will be used in oVirt is here [1].

Since our software "owns" the host networking configuration, it knows which physical interfaces belong to a bridge.  I wonder if we will have to apply these limits outside the scope of libvirt (using the operating system tools directly).

It doesn't look like your portgroup suggestion will cover our use case.  Would you agree?


[1] http://www.ovirt.org/Features/Host_Network_QoS

(In reply to Adam Litke from comment #2)
>
> Since our software "owns" the host networking configuration, it knows which
> physical interfaces belong to a bridge.  I wonder if we will have to apply
> these limits outside the scope of libvirt (using the operating system tools
> directly).
> 
> It doesn't look like your portgroup suggestion will cover our use case. 
> Would you agree?

That is correct.

(In reply to Laine Stump from comment #1)
> ...
> In the case of a <forward mode='bridge'/> network, the bridge was created by
> some other config on the system, and traffic from the guests to the rest of
> the network will likely *not* go via the bridge's own interface ('ovirtmgmt'
> in your case), so applying bandwidth controls there will do us no good.
> ...

Hi Laine,

Thanks for the thorough explanation, I just didn't get the above sentence, could you shed some light? Isn't that telling libvirt that traffic should in fact go via the bridge, thus making this a good candidate where traffic could be shaped for the entire network?

(In reply to Lior Vernia from comment #4)

> Isn't that telling libvirt that traffic should in
> fact go via the bridge, thus making this a good candidate where traffic
> could be shaped for the entire network?

No. The bandwidth limiting happens at the point where packets go in or out an interface. Packets will only go in/out the *interface* part of a bridge device if they are 1) destined for the host itself, or 2) will be routed (at L3) out of the host via its IP stack. In the case where packets are *bridged* (L2) out to the physical network, they will end up going in/out only a) the guest's tap interface and b) whatever physical interface(s) are attaching the bridge to the physical network.

So, if you want to limit the traffic between the guests and the host, you can set the limits on the bridge interface (e.g. ovirtmgmt in the example above), but if you want to limit the traffic between the guests and the physical network, then you need to set the limits on the physical interface that's attached to the bridge device (e.g. eth0, or whatever is attached to the ovirtmgmt device), in which case you could also be limiting the traffic between the *host* and the physical network (if the host also reaches the physical network via that bridge), which may or may not be what you want.

This is why I say, in the commit log message of the patch that turns this silent ignoring of <bandwidth> into an explicit error and failure, that libvirt doesn't have enough information to know what is the right thing to do.

I've pushed this patch upstream. Unless requested otherwise, I *won't* pull it into F20's libvirt, because that may break your existing configs, which you probably don't want.

In the meantime we're working on methods to support QoS for a wider array of network types, which should be in by F21 (and will almost definitely require some amount of change to the configuration that you give to libvirt).

I'm uncertain what is the best disposition for this BZ. We could mark it as CLOSED/NEXTRELEASE, to indicate that the QoS settings of a bridged network are no longer silently ignored, or we could move it to rawhide (or upstream) and keep it open in anticipation of more thorough support of different network types. Any opinions?

commit eafb53fec2436ca39949badb16b27901fe5b6ce2
Author: Laine Stump <laine>
Date:   Fri Jan 24 13:58:05 2014 +0200

    network: disallow <bandwidth>/<mac> for bridged/macvtap/hostdev networks
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1057321
    
    pointed out that we weren't honoring the <bandwidth> element in
    libvirt networks using <forward mode='bridge'/>. In fact, these
    networks are just a method of giving a libvirt network name to an
    existing Linux host bridge on the system, and libvirt doesn't have
    enough information to know where to set such limits. We are working on
    a method of supporting network bandwidths for some specific cases of
    <forward mode='bridge'/>, but currently libvirt doesn't support it. So
    the proper thing to do now is just log an error when someone tries to
    put a <bandwidth> element in that type of network. (It's unclear if we
    will be able to do proper bandwidth limiting for macvtap networks, and
    most definitely we will not be able to support it for hostdev
    networks).
    
    While looking through the network XML documentation and comparing it
    to the networkValidate function, I noticed that we also ignore the
    presence of a mac address in the config in the same cases, rather than
    failing so that the user will understand that their desired action has
    not been taken.
    
    This patch updates networkValidate() (which is called any time a
    persistent network is defined, or a transient network created) to log
    an error and fail if it finds either a <bandwidth> or <mac> element
    and the network forward mode is anything except 'route'. 'nat', or
    nothing. (Yes, neither of those elements is acceptable for any macvtap
    mode, nor for a hostdev network).
    
    NB: This does *not* cause failure to start any existing network that
    contains one of those elements, so someone might have erroneously
    defined such a network in the past, and that network will continue to
    function unmodified. I considered it too disruptive to suddenly break
    working configs on the next reboot after a libvirt upgrade.

Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-April/msg00618.html

I can not reproduce the bug, libvirt will disallow such type virtual networks, verified the bug as below:

Version:
libvirt-0.10.2-35.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.425.el6.x86_64
kernel-2.6.32-459.el6.x86_64

1. When mode='bridge', both specified bridge and device name in virtual network, libvirt will report error. 
[root@intel-3323-24-1 ~]# cat 4.xml 
<network>
<name>host_bridge</name>
<forward mode="bridge" dev="eth0"/>
<bridge name='br0' />
<bandwidth>
<inbound average='128' peak='128' burst='1024'/>
<outbound average='128' peak='128' burst='1024'/>
</bandwidth>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 4.xml 
error: Failed to define network from 4.xml
error: XML error: A network with forward mode='bridge' can specify a  bridge name or a forward dev, but not both (network 'host_bridge')

2. Disallow <mac>/<bandwidth>/<ip>/<dns>/<domian> for bridged networks
[root@intel-3323-24-1 ~]# cat 1.xml 
<network>
<name>host_bridge</name>
<forward mode='bridge'/>
<bridge name='br0' />
<bandwidth>
<inbound average='128' peak='128' burst='1024'/>
<outbound average='128' peak='128' burst='1024'/>
</bandwidth>
<mac address='52:54:00:9D:99:99'/>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 1.xml 
error: Failed to define network from 1.xml
error: unsupported configuration: Unsupported <mac> element in network host_bridge with forward mode='bridge'

[root@intel-3323-24-1 ~]# cat 1.xml 
<network>
<name>host_bridge</name>
<forward mode='bridge'/>
<bridge name='br0' />
<bandwidth>
<inbound average='128' peak='128' burst='1024'/>
<outbound average='128' peak='128' burst='1024'/>
</bandwidth>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 1.xml 
error: Failed to define network from 1.xml
error: unsupported configuration: Unsupported network-wide <bandwidth> element in network host_bridge with forward mode='bridge'

[root@intel-3323-24-1 ~]# cat 1.xml 
<network>
<name>host_bridge</name>
<forward mode='bridge'/>
<bridge name='br0' />
  <ip address="192.168.125.1" netmask="255.255.255.0">
       <dhcp>
            <range start="192.168.125.2" end="192.168.125.254" />
       </dhcp>
  </ip>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 1.xml 
error: Failed to define network from 1.xml
error: unsupported configuration: Unsupported <ip> element in network host_bridge with forward mode='bridge'

 [root@intel-3323-24-1 ~]# cat 1.xml 
<network>
<name>host_bridge</name>
<forward mode='bridge'/>
<bridge name='br0' />
<dns>
    <txt name="example" value="example value" />
    <forwarder addr="8.8.8.8"/>
    <forwarder addr="8.8.4.4"/>
    <srv service='name' protocol='tcp' domain='test-domain-name' target='.' port='1024' priority='10' weight='10'/>
    <host ip='192.168.122.2'>
    <hostname>myhost</hostname>
    <hostname>myhostalias</hostname>
    </host>
</dns>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 1.xml 
error: Failed to define network from 1.xml
error: unsupported configuration: Unsupported <dns> element in network host_bridge with forward mode='bridge'

[root@intel-3323-24-1 ~]# cat 1.xml 
<network>
<name>host_bridge</name>
<forward mode='bridge'/>
<bridge name='br0' />
<domain name="example.com"/>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 1.xml 
error: Failed to define network from 1.xml
error: unsupported configuration: Unsupported <domain> element in network host_bridge with forward mode='bridge'

3. Due to same else sentence for macvtap/hostdev network with bridged network, just tested below scenarios, others have been covered by step 2.

[root@intel-3323-24-1 ~]# cat 1.xml 
<network>
<name>hostdev</name>
<forward mode='hostdev' managed='yes'>
<driver name='vfio'/>
<address type='pci' domain='0' bus='4' slot='0' function='1'/>
<address type='pci' domain='0' bus='4' slot='0' function='2'/>
<address type='pci' domain='0' bus='4' slot='0' function='3'/>
</forward>
<mac address='52:54:00:9D:99:99'/>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 1.xml 
error: Failed to define network from 1.xml
error: unsupported configuration: Unsupported <mac> element in network hostdev with forward mode='hostdev'

[root@intel-3323-24-1 ~]# cat 2.xml 
<network>
<name>vepa-net</name>
    <forward dev='eth0' mode='vepa'>
    <interface dev='eth0'/>
    </forward>
    <mac address='52:54:00:9D:99:99'/>
    <bandwidth>
        <inbound average='128' peak='128' burst='1024'/>
        <outbound average='128' peak='128' burst='1024'/>
    </bandwidth>
</network>
[root@intel-3323-24-1 ~]# virsh net-define 2.xml 
error: Failed to define network from 2.xml
error: unsupported configuration: Unsupported <mac> element in network vepa-net with forward mode='vepa'

According to the patch series, we got the expected results.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1374.html