I don't think this is the bug of qemu which runs with unprivileged, don't even have the ability to change tc of TAP.

CC Michal for more ideas.

Well, something is clearing out QoS settings on the TAP. Mabe it's the ovs bridge then? I can see the settings applied when I step libvirt. But as soon as it spawns qemu tc settings are gone.

I have tried without libvirt,  all the pre-set rules for tap device will be flushed once it attached to the ovs bridge.

And when setting Qos rules after the tap device already attached to the ovs bridge will work.

In the scenario of comment 0, only 1 direction's rule is flushed.

# ip tuntap add  net1 mode tap
# tc qdisc add dev net1 root handle 1: htb
# tc qdisc add dev net1 ingress
# tc qdisc show dev net1
qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 0
qdisc ingress ffff: parent ffff:fff1 ---------------- 

Output for net1:
# tc class add dev net1 parent 1:0 classid 1:1 htb rate 1Mbit
# tc class show dev net1
class htb 1:1 root prio 0 rate 1000Kbit ceil 1000Kbit burst 1600b cburst 1600b 

Ingress for net1:
# tc filter add dev net1 parent ffff: protocol ip prio 50  u32 match ip src 0.0.0.0/0 police rate 2048kbps burst 1m drop flowid :1

# tc filter show dev net1 parent ffff:
filter protocol ip pref 50 u32 
filter protocol ip pref 50 u32 fh 800: ht divisor 1 
filter protocol ip pref 50 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 
  match 00000000/00000000 at 12
 police 0x7 rate 16384Kbit burst 1Mb mtu 2Kb action drop overhead 0b 
ref 1 bind 1

# ovs-vsctl show
a86f46f4-3cd7-4fef-81ce-d5942640e2ce
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "enp1s0"
            Interface "enp1s0"

# ovs-vsctl add-port ovsbr0 net1

# tc class show dev net1
#
# tc filter show dev net1 parent ffff:
#
# tc qdisc show dev net1
#

All rules will be flushed.

Set the rules again, it works

Ok. Google gives me some interesting discussion on libvirt-user

https://www.redhat.com/archives/libvirt-users/2013-July/msg00090.html
http://docs.openvswitch.org/en/latest/howto/qos/

So it looks like OVS has its own QoS mechanism. It looks to me that libvirt should switch to use OVS internal one. Any more ideas Michal?

Thanks

(In reply to jason wang from comment #5)
> Ok. Google gives me some interesting discussion on libvirt-user
> 
> https://www.redhat.com/archives/libvirt-users/2013-July/msg00090.html
> http://docs.openvswitch.org/en/latest/howto/qos/
> 
> So it looks like OVS has its own QoS mechanism. It looks to me that libvirt
> should switch to use OVS internal one. Any more ideas Michal?
> 
> Thanks

Okay, now it's more clear. Well, we can try but ideally, we would like to tell ovs that the traffic shaping rules are already set, please do not touch them. But for that I need to talk to ovs developers. Switching back to libvirt then.

Michal, do you have any updates on this subject ?

Thanks

No update here. I have a lot of bugs on my plate. Sorry.

Thanks for the reply anyway, otherwise any visibility ?

I ask because we have some (hundreds) VMs running without ingress QoS on there OVS interfaces, thus we have to make a choice here.

Thanks

I am closing this with deferred resolution. Please reopen this with justification why this should be addressed. Thanks.

The patches are merged on upstream, and the test is pass. Close the bug with resolution as upstream.
https://listman.redhat.com/archives/libvir-list/2021-August/msg00423.html

The patches and a few additional fixes are included in the upcoming 7.10.0 release.

Hi Michal, I have a question about the unit when we set bandwidth on the interface. 
According to the current status, the 'rate' related options have a unit as kilobytes/second(average, floor, peak), 'kilo' means '1000' here.
While for 'burst', which is the amount of kilobytes that can be transmitted in a single burst at peak speed, the 'kilobytes' means '1024' bytes here.
Is this expected?
If it is expected, then some fix is needed for ovs bridge type(this bug) as it use the inbound burst unit as 1000 not 1024. See below for more details:

<bandwidth>
        <inbound average='100' peak='100' burst='100'/>
        <outbound average='100' peak='200' burst='100'/>
      </bandwidth>

for outbound:
# tc -d filter show dev vnet0 parent ffff:
filter protocol all pref 49 basic chain 0 
filter protocol all pref 49 basic chain 0 handle 0x1 
 police 0x2 rate 800Kbit burst 100Kb mtu 64Kb action drop overhead 0b linklayer unspec 
	ref 1 bind 1 

for inbound:
# tc class show dev vnet0
class htb 1:1 parent 1:fffe prio 0 rate 800Kbit ceil 800Kbit burst **100000b** cburst 100000b 
class htb 1:fffe root rate 800Kbit ceil 800Kbit burst 1500b cburst 1500b

(In reply to yalzhang from comment #22)
> Hi Michal, I have a question about the unit when we set bandwidth on the
> interface. 
> According to the current status, the 'rate' related options have a unit as
> kilobytes/second(average, floor, peak), 'kilo' means '1000' here.
> While for 'burst', which is the amount of kilobytes that can be transmitted
> in a single burst at peak speed, the 'kilobytes' means '1024' bytes here.
> Is this expected?

Can I ask where do you get this from? What I can see in the documentation (https://libvirt.org/formatnetwork.html#elementQoS) is:

burst: Optional attribute which specifies the amount of kilobytes that can be transmitted in a single burst at peak speed.

Kilobytes is 1000 bytes, Kibibytes is 1024 bytes.

Hi Michal, Sorry, it's not from the documents, but from our auto test code(don't know how it was defined), and it may be wrong. As tc may show the value in different unit with different OS version, the unit conversion is defined there. So burst='100' results in 100000b is correct, I will update the auto test code.
Thank you for confirmation!

Test on libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d.x86_64

1. start vm with interface as blow:
# virsh dumpxml vm --inactive | grep /interface -B8
    <interface type='bridge'>
      <mac address='52:54:00:29:63:d5'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='226f0725-c93a-46d4-baef-4e37184789bc'/>
      </virtualport>
      <inbound average='2000' peak='3000' burst='1000'/>
        <outbound average='5000' peak='7000' burst='1000'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </interface>

for inbound:
# tc class show dev vnet0
class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst 1000000b cburst 999999b 
class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b

for outbound:
# tc -d filter show dev vnet0 parent ffff:
filter protocol all pref 49 basic chain 0 
filter protocol all pref 49 basic chain 0 handle 0x1 
 police 0x1 rate 40Mbit burst 1000Kb mtu 64Kb action drop overhead 0b linklayer unspec 
	ref 1 bind 1

The value in tc output is expected.

2. Test the actual bandwidth with netperf:
input: 22.95 10^6bits/sec
output: 40.02 10^6bits/sec

The result is as expected.

3. Adjust the bandwidth by domiftune, the result is as expected.

(In reply to Michal Privoznik from comment #23)
> (In reply to yalzhang from comment #22)
> > Hi Michal, I have a question about the unit when we set bandwidth on the
> > interface. 
> > According to the current status, the 'rate' related options have a unit as
> > kilobytes/second(average, floor, peak), 'kilo' means '1000' here.
> > While for 'burst', which is the amount of kilobytes that can be transmitted
> > in a single burst at peak speed, the 'kilobytes' means '1024' bytes here.
> > Is this expected?
> 
> Can I ask where do you get this from? What I can see in the documentation
> (https://libvirt.org/formatnetwork.html#elementQoS) is:
> 
> burst: Optional attribute which specifies the amount of kilobytes that can
> be transmitted in a single burst at peak speed.
> 
> Kilobytes is 1000 bytes, Kibibytes is 1024 bytes.

Hi Michal， from current test result for nat interface, burst use different unit(1024) with rate and peak(1000). But for ovs bridge interface, burst use the same unit with rate and peak(1000). I think it should be unified, but I don't know which one is correct. Should "inbound.burst  : 2048" be "burst 2Mb" or "burst 2000Kb"? Please help to confirm, Thank you!


# virsh domiflist rhel9
 Interface   Type      Source        Model    MAC
-----------------------------------------------------------------
 vnet15      network   default       virtio   52:54:00:dc:04:44

# virsh domiftune rhel9 vnet15
inbound.average: 2000
inbound.peak   : 3000
inbound.burst  : 2048
inbound.floor  : 0
outbound.average: 256
outbound.peak  : 512
outbound.burst : 512

# tc class show dev vnet15
class htb 1:1 root leaf 2: prio 0 rate 16Mbit ceil 24Mbit burst **2Mb** cburst 1596b 

From the tc outputs, set "inbound.average: 2000" results in rate 16Mbit, the unit is '1000'.
While for "inbound.burst  : 2048", it results in "burst 2Mb", the unit is '1024'.

But for bridge type interface connected to ovs bridge, it is different. "burst='2048'" will results in "burst 2000Kb" in tc outputs.
<interface type='bridge'>
      <mac address='52:54:00:da:b9:fb'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='67f7ce64-6ca9-4bb1-9309-f244a4e12213'/>
      </virtualport>
      <bandwidth>
        <inbound average='2000' peak='3000' burst='2048'/>
        <outbound average='256' peak='512' burst='512'/>
      </bandwidth>
      <target dev='vnet17'/>
...
# tc class show dev vnet17
class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst **2000Kb** cburst 2000Kb 
class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b

(In reply to yalzhang from comment #26)

This is the way we document burst: from https://libvirt.org/formatnetwork.html:

  burst    Optional attribute which specifies the amount of kilobytes that can be transmitted in a single burst at peak speed. 

> 
> # virsh domiflist rhel9
>  Interface   Type      Source        Model    MAC
> -----------------------------------------------------------------
>  vnet15      network   default       virtio   52:54:00:dc:04:44
> 
> # virsh domiftune rhel9 vnet15
> inbound.average: 2000
> inbound.peak   : 3000
> inbound.burst  : 2048
> inbound.floor  : 0
> outbound.average: 256
> outbound.peak  : 512
> outbound.burst : 512

Therefore, burst=2048 means 2048 kilobytes = 2048000 bytes.

> 
> # tc class show dev vnet15
> class htb 1:1 root leaf 2: prio 0 rate 16Mbit ceil 24Mbit burst **2Mb**
> cburst 1596b 
> 

This is plain stupidity of how TC displays units. 2Mb in fact is 2*1024*1024 = 2097152 bytes. Consider the following example:

# tc class add dev vnet2 parent 1:1 classid 1:2 htb rate 1000kbps burst 2097152
# tc class show dev vnet2 
class htb 1:1 root rate 8192Kbit ceil 32768Kbit burst 2Mb cburst 1593b 
class htb 1:2 parent 1:1 prio 0 rate 8Mbit ceil 8Mbit burst 2Mb cburst 1600b 

Therefore, our documentation should be fixed to mention "kibibytes" instead of kilobytes. That's what we've always configured.

> From the tc outputs, set "inbound.average: 2000" results in rate 16Mbit, the
> unit is '1000'.
> While for "inbound.burst  : 2048", it results in "burst 2Mb", the unit is
> '1024'.
> 

Yes, that's correct. Size is in multiples of 1024 (just like everywhere else in libvirt) but rate is in multiples of 1000 (= standard SI multiples), because we are no longer talking about bytes, but individual bits. While confusing that's how other tools approach the units too - just like we saw with TC a few lines above.

> But for bridge type interface connected to ovs bridge, it is different.
> "burst='2048'" will results in "burst 2000Kb" in tc outputs.
> <interface type='bridge'>
>       <mac address='52:54:00:da:b9:fb'/>
>       <source bridge='ovsbr0'/>
>       <virtualport type='openvswitch'>
>         <parameters interfaceid='67f7ce64-6ca9-4bb1-9309-f244a4e12213'/>
>       </virtualport>
>       <bandwidth>
>         <inbound average='2000' peak='3000' burst='2048'/>
>         <outbound average='256' peak='512' burst='512'/>
>       </bandwidth>
>       <target dev='vnet17'/>
> ...
> # tc class show dev vnet17
> class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst **2000Kb**
> cburst 2000Kb 
> class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b

Yeah, this should have been 2048Kb = 2Mb. Let me post a patch for that too.

Patches posted here:

https://listman.redhat.com/archives/libvir-list/2022-January/msg00094.html

Merged upstream as:

e429c062cb virnetdevopenvswitch: Fix 'burst' value passed to ovs-vsctl
f3b422d9cc docs: Clarify 'burst' units for QoS

v7.10.0-413-ge429c062cb

For comment 29, test on libvirt-8.0.0-0rc1.1.el9.x86_64, the result is expected.
# virsh dumpxml rh9 | grep /interface -B14
    <interface type='bridge'>
      <mac address='52:54:00:da:b9:fb'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='67f7ce64-6ca9-4bb1-9309-f244a4e12213'/>
      </virtualport>
      <bandwidth>
        <inbound average='2000' peak='3000' burst='2048'/>
        <outbound average='256' peak='512' burst='512'/>
      </bandwidth>
      <target dev='vnet58'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:53:be:14'/>
      <source network='default' portid='8b55c007-9d0a-49c9-82db-6b479ee5f415' bridge='virbr0'/>
      <bandwidth>
        <inbound average='2000' peak='3000' burst='2048'/>
        <outbound average='256' peak='512' burst='512'/>
      </bandwidth>
      <target dev='vnet59'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </interface>

[root@sriov ~]# tc class show dev vnet59
class htb 1:1 root leaf 2: prio 0 rate 16Mbit ceil 24Mbit burst 2Mb cburst 1596b 
[root@sriov ~]# 
[root@sriov ~]# tc class show dev vnet58
class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst 2Mb cburst 2Mb 
class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1759