Bug 976648 - Issues with inbound and outbound elements under bandwidth and peak attribute of outbound element
Issues with inbound and outbound elements under bandwidth and peak attribute ...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Michal Privoznik
Virtualization Bugs
:
Depends On:
Blocks: 980350
  Show dependency treegraph
 
Reported: 2013-06-21 01:57 EDT by hongming
Modified: 2014-06-17 20:51 EDT (History)
6 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 980350 (view as bug list)
Environment:
Last Closed: 2014-06-13 08:07:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description hongming 2013-06-21 01:57:31 EDT
Description of problem:
Issue 1 - The vnetX (virtual interface) is regarded as one part of domain , the inbound (domain's input or download ) in xml should use policing (tc filter on vnet0) and the outbound (domain' output or upload )  should use shaping (tc class on vnetX) .  But libvirt use 'tc' in reverse . please refer to the following steps. 

Issue 2 - The peak attribute of outbound element is useless for now It is as policing in linux kernel doesn't allow to set anything like that. But according to libvirt.org, it specifies maximum rate at which interface can send data , we can remove it from xml schema or add explanation in document. 

The issues will confuse user who don't know tc and libvirt code even if it don't affect the network QoS.

Version-Release number of selected component (if applicable):
kernel-3.10.0-0.rc4.59.el7.x86_64
libvirt-1.0.6-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
# brctl show
bridge name    bridge id        STP enabled    interfaces
virbr0        8000.525400af8736    yes         virbr0-nic
                                               vnet0

# virsh dumpxml rhel7|grep bandwidth -A 1
      <bandwidth>
        <outbound average='5000' peak='2000' burst='1024'/> <====== The peak ='2000' is useless. but according to document , it specifies maximum rate at which interface can send data
      </bandwidth>


# tc -d filter show dev vnet0 parent ffff:
filter protocol ip pref 49152 u32
filter protocol ip pref 49152 u32 fh 800: ht divisor 1
filter protocol ip pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1
  match 00000000/00000000 at 12
 police 0x5 rate 40000Kbit burst 1024Kb mtu 64Kb action drop overhead 0b  


# virsh edit rhel7  (Change outbound to inbound)
Domain rhel7 XML configuration edited.

# virsh destroy rhel7
Domain rhel7 destroyed

# virsh start rhel7
Domain rhel7 started

# virsh dumpxml rhel7|grep bandwidth -A 1
      <bandwidth>
        <inbound average='5000' peak='2000' burst='1024'/>
      </bandwidth>

# tc -d class show dev vnet0
class htb 1:1 root leaf 2: prio 0 quantum 200000 rate 40000Kbit ceil 16000Kbit burst 1024Kb/1 mpu 0b overhead 0b cburst 419428b/1 mpu 0b overhead 0b level 0

Document as follows
http://libvirt.org/formatdomain.html#elementQoS
Incoming and outgoing traffic can be shaped independently. The bandwidth element can have at most one inbound and at most one outbound child elements. Leaving any of these children element out result in no QoS applied on that traffic direction. So, when you want to shape only domain's incoming traffic, use inbound only, and vice versa. Each of these elements have one mandatory attribute average (or floor as described below). average specifies average bit rate on the interface being shaped. Then there are two optional attributes: peak, which specifies maximum rate at which interface can send data, and burst, amount of bytes that can be burst at peak speed.

Actual results:
as above

Expected results:
The inbound in xml should use policing (tc filter on vnet0) and  the outbound should use shaping (tc class on vnetX)
The peak attribute of outbound element from xml schema or add explanation in document . 

Additional info:
Comment 2 Michal Privoznik 2013-07-26 06:05:35 EDT
The Issue 1 is not an issue. The current code is right. Domain's incoming traffic is host's outgoing. And domain's outgoing traffic is host's incoming. Moreover, the linux kernel allows us to shape outgoing traffic, and 'just' police incoming. This makes sense - a host can't influence how much traffic is sent to it. If somebody on the network decides to send huge amount of packets to our host, there's nothing we can do about it. I mean, we can drop the packets, but the packets will still be delivered. The situation is different with our host originated packets. The kernel can block write() to NIC's buffer to meet desired packet rate. Having said that, the Issue 1 is not an issue.

For the second issue, I've just pushed the patch which will be picked up by rebase:

commit d7a4a9b2ffe134eaf33c1965ab29a0fae416e166
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Fri Jul 26 11:42:53 2013 +0200
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Fri Jul 26 11:45:10 2013 +0200

    formatdomain.html.in: Document implementation limitation of QoS
    
    The outbound/@peak is ignored (since QoS was introduced). This is due to
    kernel limitation of know allowing ingress filters to have peak just
    average rate. However, we should document this limitation to not confuse
    users.

v1.1.1-rc1-33-gd7a4a9b
Comment 3 hongming 2013-07-28 23:37:06 EDT
(In reply to Michal Privoznik from comment #2)
> The Issue 1 is not an issue. The current code is right. Domain's incoming
> traffic is host's outgoing. And domain's outgoing traffic is host's
> incoming. Moreover, the linux kernel allows us to shape outgoing traffic,
> and 'just' police incoming. This makes sense - a host can't influence how
> much traffic is sent to it. If somebody on the network decides to send huge
> amount of packets to our host, there's nothing we can do about it. I mean,
> we can drop the packets, but the packets will still be delivered. The
> situation is different with our host originated packets. The kernel can
> block write() to NIC's buffer to meet desired packet rate. Having said that,
> the Issue 1 is not an issue.

Hi Michal 

Is the vnet0(virtual interface,tap device) regarded as one part of guest or one part of host?

If the vnet0(virtual interface) is regarded as one part of guest, the inbound (domain's input or download) in xml should use policing (tc filter on vnet0) and  the outbound (domain'output or upload) should use shaping (tc class on vnet0). Now the actual result is opposite.


And when we only config bandwidth for virbr0(virtual network switch), according to libvirt.org, we only use the inbound in network'xml to shape only network's incoming traffic. and vice versa. But the following actual result is opposite.

"So, when you want to shape only network's incoming traffic, use inbound only, and vice versa." - libvirt.org


Actual result: 
# virsh net-dumpxml default|grep bandwidth -A 1
  <bandwidth>
    <inbound average='5000' peak='2000' burst='1024'/>  
  </bandwidth>                                           <===== direction : packets --> NIC---> virbr0---> vnet0 

# tc -d filter show dev virbr0 parent ffff:              <===== no tc filter be created on virbr0

# tc -d class show dev virbr0                            
class htb 1:1 root rate 40000Kbit ceil 16000Kbit burst 1600b/8 mpu 0b overhead 0b cburst 1600b/8 mpu 0b overhead 0b level 7 
class htb 1:2 parent 1:1 leaf 2: prio 0 quantum 200000 rate 40000Kbit ceil 16000Kbit burst 1024Kb/8 mpu 0b overhead 0b cburst 1600b/8 mpu 0b overhead 0b level 0                                                  <===== Create tc class on virbr0 . 


Thanks


> For the second issue, I've just pushed the patch which will be picked up by
> rebase:
> 
> commit d7a4a9b2ffe134eaf33c1965ab29a0fae416e166
> Author:     Michal Privoznik <mprivozn@redhat.com>
> AuthorDate: Fri Jul 26 11:42:53 2013 +0200
> Commit:     Michal Privoznik <mprivozn@redhat.com>
> CommitDate: Fri Jul 26 11:45:10 2013 +0200
> 
>     formatdomain.html.in: Document implementation limitation of QoS
>     
>     The outbound/@peak is ignored (since QoS was introduced). This is due to
>     kernel limitation of know allowing ingress filters to have peak just
>     average rate. However, we should document this limitation to not confuse
>     users.
> 
> v1.1.1-rc1-33-gd7a4a9b
Comment 4 hongming 2013-07-31 03:19:19 EDT
Verify it as follows. Move its status to VERIFIED.

# vim /usr/share/doc/libvirt-docs-1.1.1/html/formatdomain.html

3151 Note the limitation of implementation: the <code>peak</code> attribute in
3152 <code>outbound</code> element is ignored (as linux ingress filters don't
3153 know it yet). <span class="since">Since 0.9.4</span> The <code>inbound</code> can
Comment 5 Ludek Smid 2014-06-13 08:07:29 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.