Bug 1497410

Patches proposed on the upstream list:

https://www.redhat.com/archives/libvir-list/2017-October/msg00026.html

I've pushed the patches online:

commit 57b17c8d84f237741f14ad988746bb1956c459c3
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Oct 2 16:31:30 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Oct 5 09:16:05 2017 +0200

    libvirt-domain: Document interface stats POV
    
    Interestingly enough, we don't document the point of view of the
    interface statistics. Therefore it's unknown to users if for
    instance rx_packets is the number of packets received by domain or
    received by host (from domain). Document this explicitly.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit cea3715b2e9a8faf9ed3a6d6699c03b52cfa769b
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Oct 2 14:12:44 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Oct 5 09:16:05 2017 +0200

    QoS: Set classes and filters in proper direction
    
    Similarly to previous patch, for some types of interface domain
    and host are on the same side of RX/TX barrier. In that case, we
    need to set up the QoS differently. Well, swapped.
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: John Ferlan <jferlan>

commit d86fd2402e9d217f90e0a149d6486f392b34f7ba
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Oct 2 13:36:56 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Oct 5 09:16:05 2017 +0200

    virNetDevTapInterfaceStats: Allow caller to not swap the statistics
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1497410
    
    The comment in virNetDevTapInterfaceStats() implementation for
    Linux states that packets transmitted by domain are received by
    the host and vice versa. Well, this is true but not for all types
    of interfaces. For instance, for macvtaps when TAP device is
    hooked right onto a physical device any packet that domain sends
    looks also like a packet sent to the host. Therefore, we should
    allow caller to chose if the stats returned should be straight
    copy or swapped.
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: John Ferlan <jferlan>

v3.8.0-20-g57b17c8d8

build upstream libvirt as libvirt-3.9.0-1.el7.x86_64, test with below scenarios, all the results are as expected for direct type interface.

For domifstat with direct type interface, the result is as expected.
1. start a guest with direct type interface
# virsh dumpxml rhel7.4 | grep /interface -B7
    <interface type='direct'>
      <mac address='52:54:00:08:ae:94'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
2. login guest and ping somewhere to get statistics, then check on guest and host
on guest:
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.66.70.120 netmask 255.255.254.0 broadcast 10.66.71.255
        inet6 fe80::5054:ff:fe08:ae94 prefixlen 64 scopeid 0x20<link>
        inet6 2620:52:0:4246:5054:ff:fe08:ae94 prefixlen 64 scopeid 0x0<global>
        ether 52:54:00:08:ae:94 txqueuelen 1000 (Ethernet)
        RX packets 206 bytes 31096 (30.3 KiB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 94 bytes 8745 (8.5 KiB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
on host:
# virsh domifstat rhel7.4 macvtap0
macvtap0 rx_bytes 30734
macvtap0 rx_packets 207
macvtap0 rx_errs 0
macvtap0 rx_drop 0
macvtap0 tx_bytes 9435
macvtap0 tx_packets 103
macvtap0 tx_errs 0
macvtap0 tx_drop 0

The domifstat outputs shows guest point of view, here it is reasonable compared with the outputs of ifconfig on guest.

For Qos issue, the result is as expected
1. start a guest with Qos setting in direct type interface
# virsh dumpxml rhel7.4 | grep /interface -B11
    <interface type='direct'>
      <mac address='52:54:00:08:ae:94'/>
      <source dev='eno1' mode='bridge'/>
      <bandwidth>
        <inbound average='100' peak='200' burst='128'/>
        <outbound average='50' peak='100' burst='128'/>
      </bandwidth>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2. check the rules by tc
# tc filter show dev macvtap0 parent ffff:   ===> inbound in xml
filter protocol all pref 49152 u32 
filter protocol all pref 49152 u32 fh 800: ht divisor 1 
filter protocol all pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 
  match 00000000/00000000 at 0
 police 0x1 rate 800000bit burst 128Kb mtu 64Kb action drop overhead 0b 
ref 1 bind 1

# tc class show dev macvtap0   ===> outbound in xml
class htb 1:1 root leaf 2: prio 0 rate 400000bit ceil 800000bit burst 128Kb cburst 1600b 

3. test guest's inbound(expected result: around 0.8Mbit/s)
on guest: # netserver
on another host
# netperf -H 10.66.4.161 -l 60
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.4.161 () port 0 AF_INET
Recv Send Send                          
Socket Socket Message Elapsed              
Size Size Size Time Throughput  
bytes bytes bytes secs. 10^6bits/sec  

 87380 16384 16384 61.15 0.80   

4. test guest's outbound(expected result: around 0.4Mbit/s)
on another host: # netserver
on guest:
# netperf -H 10.66.5.24 -l 60
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.66.5.24 () port 0 AF_INET
Recv Send Send                          
Socket Socket Message Elapsed              
Size Size Size Time Throughput  
bytes bytes bytes secs. 10^6bits/sec  

 87380 16384 16384 64.00 0.40

Hi Michal, 

For direct type interface, all works well as above comment shows. 
For bridge type interface with ovs, as I have tested with above patches, when start the vm, the statistics for outbound still will be swapped.
But when set Qos on the fly by domiftune, it works well.


1. prepare an inactive domain with bridge type interface connected to ovs
# virsh dumpxml rhel7.4 | grep /interface -B12
    <interface type='bridge'>
      <mac address='52:54:00:f1:87:de'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='736e1c04-1274-4f68-beec-18e1fbd95bfa'/>
      </virtualport>
      <bandwidth>
        <inbound average='200' peak='300' burst='128'/>
        <outbound average='100' peak='200' burst='128'/>
      </bandwidth>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2.# virsh start rhel7.4
Domain rhel7.4 started

# virsh domiflist rhel7.4 
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     ovsbr0     rtl8139     52:54:00:f1:87:de

# tc class show dev vnet0
class htb 1:1 root leaf 2: prio 0 rate 1600Kbit ceil 2400Kbit burst 128Kb cburst 1599b 

# tc filter show dev vnet0 parent ffff:
===> no output

But when set Qos on the fly, it works

1. start the vm with no Qos setting
# virsh start rhel7.4

# virsh dumpxml rhel7.4 | grep /interface -B10
    <interface type='bridge'>
      <mac address='52:54:00:f1:87:de'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='736e1c04-1274-4f68-beec-18e1fbd95bfa'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2. Set Qos on the fly, it works well
# virsh domiftune rhel7.4 vnet0 --inbound 200,300,128 --outbound 100,200,128

# virsh domiftune rhel7.4 vnet0
inbound.average: 200
inbound.peak   : 300
inbound.burst  : 128
inbound.floor  : 0
outbound.average: 100
outbound.peak  : 200
outbound.burst : 128

3. check the rules
# tc filter show dev vnet0 parent ffff:
filter protocol all pref 49152 u32 
filter protocol all pref 49152 u32 fh 800: ht divisor 1 
filter protocol all pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 
  match 00000000/00000000 at 0
 police 0x9 rate 800000bit burst 128Kb mtu 64Kb action drop overhead 0b 
ref 1 bind 1

# tc class show dev vnet0
class htb 1:1 root leaf 2: prio 0 rate 1600Kbit ceil 2400Kbit burst 128Kb cburst 1599b 

# virsh dumpxml rhel7.4 | grep /interface -B9
      </virtualport>
      <bandwidth>
        <inbound average='200' peak='300' burst='128'/>
        <outbound average='100' peak='200' burst='128'/>
      </bandwidth>
...

4. test bandwidth, for guest, inbound: 1.53Mbit/s, outboud: 0.8Mbit/s, all things are expected.

(In reply to yalzhang from comment #5)
> Hi Michal, 
> 
> For direct type interface, all works well as above comment shows. 
> For bridge type interface with ovs, as I have tested with above patches,
> when start the vm, the statistics for outbound still will be swapped.
> But when set Qos on the fly by domiftune, it works well.
> 

This is a separate bug. I've proposed patches for that here:

https://www.redhat.com/archives/libvir-list/2017-October/msg01144.html

This one is fixed IMO.

I've pushed the referenced patches upstream:

commit bc8a99ef06417a2303ccab455f9f045e2a617916
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Oct 25 12:45:03 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 27 14:59:29 2017 +0200

    virNetDevSupportBandwidth: Enable QoS for vhostuser
    
    Since vhostuser type is really a tap that is just plugged into
    different type of bridge, supporting QoS is trivial.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 98696fd170b7ff4ecd4b8499ee1952f11cb26e46
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Oct 25 12:42:41 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Oct 27 14:59:29 2017 +0200

    qemuDomainSetInterfaceParameters: Explicitly reject unsupported net types
    
    For instance, NET_TYPE_MCAST doesn't support setting QoS. Instead
    of claiming success and doing nothing, we should be explicit
    about that and report an error.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v3.9.0-rc1-2-gbc8a99ef0

Hi Michal, I have tested on libvirt-3.9.0-1.el7.x86_64, with the patches on comment 7, the issue in comment 5 still happens.

For the 2nd patch, I test with other interface types, no error message will popup, guest start successfully with bandwidth, but has no function. 

For vhostuser, seems different from tap device to support Qos...


Details:
1. Start a vm with bridge type interface with Qos setting which connected to ovs, there is no ingress qdisc created just as comment 5.

# tc -d qdisc show dev vnet0 
qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 ver 3.17 direct_qlen 1000
qdisc sfq 2: parent 1:1 limit 127p quantum 1514b depth 127 flows 128/1024 divisor 1024 perturb 10sec 

===> no ingress qdisc created

2. Guest can start without error when set bandwidth in mcast user type interfaces, the Qos settting is invalid.
# virsh edit rhel7
...
  <interface type='mcast'>
      <mac address='52:54:00:6d:90:01'/>
      <source address='230.0.0.1' port='5558'/>
      <bandwidth>
        <inbound average='1000' peak='5000' burst='1024'/>
        <outbound average='128' peak='256' burst='256'/>
      </bandwidth>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='user'>
      <mac address='00:11:22:33:44:55'/>
      <bandwidth>
        <inbound average='1000' peak='5000' burst='1024'/>
        <outbound average='128' peak='256' burst='256'/>
      </bandwidth>
      <ip address='172.17.2.0' family='ipv4' prefix='24'/>
      <ip address='2001:db8:ac10:fd01::' family='ipv6' prefix='64'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
...
# virsh start rhel7
Domain rhel7 started

3. For vhostuser, no tc rules found, or I deal it in the wrong way? 
# virsh edit vhost1
...
  <interface type='vhostuser'>
      <mac address='52:54:00:69:33:29'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <bandwidth>
        <inbound average='1000' peak='5000' burst='1024'/>
        <outbound average='128' peak='256' burst='256'/>
      </bandwidth>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
..
# virsh start vhost1
Domain vhost1 started

# virsh domiflist vhost1
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vhost-user1 vhostuser  -          virtio      52:54:00:69:33:29

# ovs-vsctl show
83e0fab0-0bb6-4e80-8749-a41c7b702c32
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser

# tc qdisc show dev vhost-user2 
Cannot find device "vhost-user2"

(In reply to yalzhang from comment #9)
> Hi Michal, I have tested on libvirt-3.9.0-1.el7.x86_64, with the patches on
> comment 7, the issue in comment 5 still happens.
> 
> For the 2nd patch, I test with other interface types, no error message will
> popup, guest start successfully with bandwidth, but has no function. 
> 
> For vhostuser, seems different from tap device to support Qos...
> 
> 
> Details:
> 1. Start a vm with bridge type interface with Qos setting which connected to
> ovs, there is no ingress qdisc created just as comment 5.
> 
> # tc -d qdisc show dev vnet0 
> qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 ver 3.17
> direct_qlen 1000
> qdisc sfq 2: parent 1:1 limit 127p quantum 1514b depth 127 flows 128/1024
> divisor 1024 perturb 10sec 
> 
> ===> no ingress qdisc created

I've debugged the problem and it looks like libvirt sets QoS rules and they remain set until qemu takes over the interface. When qemu is initializing it clears out previous settings set by libvirt. I suggest filing a bug against qemu.

> 
> 2. Guest can start without error when set bandwidth in mcast user type
> interfaces, the Qos settting is invalid.

Yes. But a warning is produced. This is a workaround for broken configurations. Back in the day when QoS was introduced I made it so that domains with QoS defined for unsupported devices can start (QoS setting was simply ignored). To not break something we have to preserve this behaviour. Unfortunately. So if you see the warning message in logs this is expected behaviour (not that I like it).

> # virsh edit rhel7
> ...
>   <interface type='mcast'>
>       <mac address='52:54:00:6d:90:01'/>
>       <source address='230.0.0.1' port='5558'/>
>       <bandwidth>
>         <inbound average='1000' peak='5000' burst='1024'/>
>         <outbound average='128' peak='256' burst='256'/>
>       </bandwidth>
>       <model type='rtl8139'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0'/>
>     </interface>
>     <interface type='user'>
>       <mac address='00:11:22:33:44:55'/>
>       <bandwidth>
>         <inbound average='1000' peak='5000' burst='1024'/>
>         <outbound average='128' peak='256' burst='256'/>
>       </bandwidth>
>       <ip address='172.17.2.0' family='ipv4' prefix='24'/>
>       <ip address='2001:db8:ac10:fd01::' family='ipv6' prefix='64'/>
>       <model type='rtl8139'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>     </interface>
> ...
> # virsh start rhel7
> Domain rhel7 started
> 
> 3. For vhostuser, no tc rules found, or I deal it in the wrong way? 
> # virsh edit vhost1
> ...
>   <interface type='vhostuser'>
>       <mac address='52:54:00:69:33:29'/>
>       <source type='unix' path='/var/run/openvswitch/vhost-user1'
> mode='client'/>
>       <bandwidth>
>         <inbound average='1000' peak='5000' burst='1024'/>
>         <outbound average='128' peak='256' burst='256'/>
>       </bandwidth>
>       <model type='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
> ..
> # virsh start vhost1
> Domain vhost1 started
> 
> # virsh domiflist vhost1
> Interface  Type       Source     Model       MAC
> -------------------------------------------------------
> vhost-user1 vhostuser  -          virtio      52:54:00:69:33:29
> 
> # ovs-vsctl show
> 83e0fab0-0bb6-4e80-8749-a41c7b702c32
>     Bridge "ovsbr0"
>         Port "ovsbr0"
>             Interface "ovsbr0"
>                 type: internal
>         Port "vhost-user2"
>             Interface "vhost-user2"
>                 type: dpdkvhostuser
>         Port "vhost-user1"
>             Interface "vhost-user1"
>                 type: dpdkvhostuser
> 
> # tc qdisc show dev vhost-user2 
> Cannot find device "vhost-user2"

This needs to be vhost-user1.

(In reply to Michal Privoznik from comment #10)
 
> I've debugged the problem and it looks like libvirt sets QoS rules and they
> remain set until qemu takes over the interface. When qemu is initializing it
> clears out previous settings set by libvirt. I suggest filing a bug against
> qemu.

Submit a bug 1510237

> > 2. Guest can start without error when set bandwidth in mcast user type
> > interfaces, the Qos settting is invalid.
> 
> Yes. But a warning is produced. This is a workaround for broken
> configurations. Back in the day when QoS was introduced I made it so that
> domains with QoS defined for unsupported devices can start (QoS setting was
> simply ignored). To not break something we have to preserve this behaviour.
> Unfortunately. So if you see the warning message in logs this is expected
> behaviour (not that I like it).

Thank you! I got the warning message. It's enough, and I like it, too.
# cat /var/log/libvirt/libvirtd.log | grep warning
2017-11-07 01:14:02.604+0000: 1337: warning : qemuBuildInterfaceCommandLine:8630 : setting bandwidth on interfaces of type 'mcast' is not implemented yet
2017-11-07 01:14:02.604+0000: 1337: warning : qemuBuildInterfaceCommandLine:8630 : setting bandwidth on interfaces of type 'user' is not implemented yet

 
> > # virsh domiflist vhost1
> > Interface  Type       Source     Model       MAC
> > -------------------------------------------------------
> > vhost-user1 vhostuser  -          virtio      52:54:00:69:33:29
> > 
> > # ovs-vsctl show
> > 83e0fab0-0bb6-4e80-8749-a41c7b702c32
> >     Bridge "ovsbr0"
> >         Port "ovsbr0"
> >             Interface "ovsbr0"
> >                 type: internal
> >         Port "vhost-user2"
> >             Interface "vhost-user2"
> >                 type: dpdkvhostuser
> >         Port "vhost-user1"
> >             Interface "vhost-user1"
> >                 type: dpdkvhostuser
> > 
> > # tc qdisc show dev vhost-user2 
> > Cannot find device "vhost-user2"
> 
> This needs to be vhost-user1.

Sorry for the mistake. I redo the test with vhost-user1, but the result is the same. I doubted if tc only apply rules for those interfaces which can be listed by "#ip link show"? the source of the vhostuser interface here I use is a unix socket.

<source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>

(In reply to yalzhang from comment #11)

> > > # virsh domiflist vhost1
> > > Interface  Type       Source     Model       MAC
> > > -------------------------------------------------------
> > > vhost-user1 vhostuser  -          virtio      52:54:00:69:33:29
> > > 
> > > # ovs-vsctl show
> > > 83e0fab0-0bb6-4e80-8749-a41c7b702c32
> > >     Bridge "ovsbr0"
> > >         Port "ovsbr0"
> > >             Interface "ovsbr0"
> > >                 type: internal
> > >         Port "vhost-user2"
> > >             Interface "vhost-user2"
> > >                 type: dpdkvhostuser
> > >         Port "vhost-user1"
> > >             Interface "vhost-user1"
> > >                 type: dpdkvhostuser
> > > 
> > > # tc qdisc show dev vhost-user2 
> > > Cannot find device "vhost-user2"
> > 
> > This needs to be vhost-user1.
> 
> Sorry for the mistake. I redo the test with vhost-user1, but the result is
> the same. I doubted if tc only apply rules for those interfaces which can be
> listed by "#ip link show"? the source of the vhostuser interface here I use
> is a unix socket.
> 
> <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>

Oh right. But vhostuser can only have <source type="unix"/>. I wonder what I was thinking when writing bc8a99ef064. vhostuser (=dpdk enabled interface) does not support QoS. However, tap devices plugged into a ovs bridge do support it. But they are type="bridge" not "vhostuser". I need to revert that patch.

Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2017-November/msg00393.html

And here's the backport:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2017-November/msg00323.html

And scratch build:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14538703

Test on libvirt-3.9.0-2.el7.x86_64, the result is as expected, only one small issue that vhostuser type interface with Qos setting do not get the error message, have informed devel and he said will fix it upstream. Set this bug to be verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704