Bug 1510237
Summary: | Libvirt should use ovs-vsctl for setting QoS for TAPs plugged into ovs. | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | yalzhang <yalzhang> |
Component: | libvirt | Assignee: | Virtualization Maintenance <virt-maint> |
Status: | CLOSED ERRATA | QA Contact: | yalzhang <yalzhang> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 8.0 | CC: | ailan, chayang, Egarciad, jdenemar, jsuchane, juzhang, knoel, lmen, molasaga, mprivozn, pezhang, rbalakri, virt-maint, xuzhang, yipikai7 |
Target Milestone: | rc | Keywords: | Reopened, Upstream |
Target Release: | 8.6 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2022-05-10 13:18:34 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | 7.10.0 |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1381612 |
Description
yalzhang@redhat.com
2017-11-07 01:07:33 UTC
I don't think this is the bug of qemu which runs with unprivileged, don't even have the ability to change tc of TAP. CC Michal for more ideas. Well, something is clearing out QoS settings on the TAP. Mabe it's the ovs bridge then? I can see the settings applied when I step libvirt. But as soon as it spawns qemu tc settings are gone. I have tried without libvirt, all the pre-set rules for tap device will be flushed once it attached to the ovs bridge. And when setting Qos rules after the tap device already attached to the ovs bridge will work. In the scenario of comment 0, only 1 direction's rule is flushed. # ip tuntap add net1 mode tap # tc qdisc add dev net1 root handle 1: htb # tc qdisc add dev net1 ingress # tc qdisc show dev net1 qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 0 qdisc ingress ffff: parent ffff:fff1 ---------------- Output for net1: # tc class add dev net1 parent 1:0 classid 1:1 htb rate 1Mbit # tc class show dev net1 class htb 1:1 root prio 0 rate 1000Kbit ceil 1000Kbit burst 1600b cburst 1600b Ingress for net1: # tc filter add dev net1 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 2048kbps burst 1m drop flowid :1 # tc filter show dev net1 parent ffff: filter protocol ip pref 50 u32 filter protocol ip pref 50 u32 fh 800: ht divisor 1 filter protocol ip pref 50 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 match 00000000/00000000 at 12 police 0x7 rate 16384Kbit burst 1Mb mtu 2Kb action drop overhead 0b ref 1 bind 1 # ovs-vsctl show a86f46f4-3cd7-4fef-81ce-d5942640e2ce Bridge "ovsbr0" Port "ovsbr0" Interface "ovsbr0" type: internal Port "enp1s0" Interface "enp1s0" # ovs-vsctl add-port ovsbr0 net1 # tc class show dev net1 # # tc filter show dev net1 parent ffff: # # tc qdisc show dev net1 # All rules will be flushed. Set the rules again, it works Ok. Google gives me some interesting discussion on libvirt-user https://www.redhat.com/archives/libvirt-users/2013-July/msg00090.html http://docs.openvswitch.org/en/latest/howto/qos/ So it looks like OVS has its own QoS mechanism. It looks to me that libvirt should switch to use OVS internal one. Any more ideas Michal? Thanks (In reply to jason wang from comment #5) > Ok. Google gives me some interesting discussion on libvirt-user > > https://www.redhat.com/archives/libvirt-users/2013-July/msg00090.html > http://docs.openvswitch.org/en/latest/howto/qos/ > > So it looks like OVS has its own QoS mechanism. It looks to me that libvirt > should switch to use OVS internal one. Any more ideas Michal? > > Thanks Okay, now it's more clear. Well, we can try but ideally, we would like to tell ovs that the traffic shaping rules are already set, please do not touch them. But for that I need to talk to ovs developers. Switching back to libvirt then. Michal, do you have any updates on this subject ? Thanks No update here. I have a lot of bugs on my plate. Sorry. Thanks for the reply anyway, otherwise any visibility ? I ask because we have some (hundreds) VMs running without ingress QoS on there OVS interfaces, thus we have to make a choice here. Thanks I am closing this with deferred resolution. Please reopen this with justification why this should be addressed. Thanks. The patches are merged on upstream, and the test is pass. Close the bug with resolution as upstream. https://listman.redhat.com/archives/libvir-list/2021-August/msg00423.html The patches and a few additional fixes are included in the upcoming 7.10.0 release. Hi Michal, I have a question about the unit when we set bandwidth on the interface. According to the current status, the 'rate' related options have a unit as kilobytes/second(average, floor, peak), 'kilo' means '1000' here. While for 'burst', which is the amount of kilobytes that can be transmitted in a single burst at peak speed, the 'kilobytes' means '1024' bytes here. Is this expected? If it is expected, then some fix is needed for ovs bridge type(this bug) as it use the inbound burst unit as 1000 not 1024. See below for more details: <bandwidth> <inbound average='100' peak='100' burst='100'/> <outbound average='100' peak='200' burst='100'/> </bandwidth> for outbound: # tc -d filter show dev vnet0 parent ffff: filter protocol all pref 49 basic chain 0 filter protocol all pref 49 basic chain 0 handle 0x1 police 0x2 rate 800Kbit burst 100Kb mtu 64Kb action drop overhead 0b linklayer unspec ref 1 bind 1 for inbound: # tc class show dev vnet0 class htb 1:1 parent 1:fffe prio 0 rate 800Kbit ceil 800Kbit burst **100000b** cburst 100000b class htb 1:fffe root rate 800Kbit ceil 800Kbit burst 1500b cburst 1500b (In reply to yalzhang from comment #22) > Hi Michal, I have a question about the unit when we set bandwidth on the > interface. > According to the current status, the 'rate' related options have a unit as > kilobytes/second(average, floor, peak), 'kilo' means '1000' here. > While for 'burst', which is the amount of kilobytes that can be transmitted > in a single burst at peak speed, the 'kilobytes' means '1024' bytes here. > Is this expected? Can I ask where do you get this from? What I can see in the documentation (https://libvirt.org/formatnetwork.html#elementQoS) is: burst: Optional attribute which specifies the amount of kilobytes that can be transmitted in a single burst at peak speed. Kilobytes is 1000 bytes, Kibibytes is 1024 bytes. Hi Michal, Sorry, it's not from the documents, but from our auto test code(don't know how it was defined), and it may be wrong. As tc may show the value in different unit with different OS version, the unit conversion is defined there. So burst='100' results in 100000b is correct, I will update the auto test code. Thank you for confirmation! Test on libvirt-7.10.0-1.module+el8.6.0+13502+4f24a11d.x86_64 1. start vm with interface as blow: # virsh dumpxml vm --inactive | grep /interface -B8 <interface type='bridge'> <mac address='52:54:00:29:63:d5'/> <source bridge='ovsbr0'/> <virtualport type='openvswitch'> <parameters interfaceid='226f0725-c93a-46d4-baef-4e37184789bc'/> </virtualport> <inbound average='2000' peak='3000' burst='1000'/> <outbound average='5000' peak='7000' burst='1000'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </interface> for inbound: # tc class show dev vnet0 class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst 1000000b cburst 999999b class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b for outbound: # tc -d filter show dev vnet0 parent ffff: filter protocol all pref 49 basic chain 0 filter protocol all pref 49 basic chain 0 handle 0x1 police 0x1 rate 40Mbit burst 1000Kb mtu 64Kb action drop overhead 0b linklayer unspec ref 1 bind 1 The value in tc output is expected. 2. Test the actual bandwidth with netperf: input: 22.95 10^6bits/sec output: 40.02 10^6bits/sec The result is as expected. 3. Adjust the bandwidth by domiftune, the result is as expected. (In reply to Michal Privoznik from comment #23) > (In reply to yalzhang from comment #22) > > Hi Michal, I have a question about the unit when we set bandwidth on the > > interface. > > According to the current status, the 'rate' related options have a unit as > > kilobytes/second(average, floor, peak), 'kilo' means '1000' here. > > While for 'burst', which is the amount of kilobytes that can be transmitted > > in a single burst at peak speed, the 'kilobytes' means '1024' bytes here. > > Is this expected? > > Can I ask where do you get this from? What I can see in the documentation > (https://libvirt.org/formatnetwork.html#elementQoS) is: > > burst: Optional attribute which specifies the amount of kilobytes that can > be transmitted in a single burst at peak speed. > > Kilobytes is 1000 bytes, Kibibytes is 1024 bytes. Hi Michal, from current test result for nat interface, burst use different unit(1024) with rate and peak(1000). But for ovs bridge interface, burst use the same unit with rate and peak(1000). I think it should be unified, but I don't know which one is correct. Should "inbound.burst : 2048" be "burst 2Mb" or "burst 2000Kb"? Please help to confirm, Thank you! # virsh domiflist rhel9 Interface Type Source Model MAC ----------------------------------------------------------------- vnet15 network default virtio 52:54:00:dc:04:44 # virsh domiftune rhel9 vnet15 inbound.average: 2000 inbound.peak : 3000 inbound.burst : 2048 inbound.floor : 0 outbound.average: 256 outbound.peak : 512 outbound.burst : 512 # tc class show dev vnet15 class htb 1:1 root leaf 2: prio 0 rate 16Mbit ceil 24Mbit burst **2Mb** cburst 1596b From the tc outputs, set "inbound.average: 2000" results in rate 16Mbit, the unit is '1000'. While for "inbound.burst : 2048", it results in "burst 2Mb", the unit is '1024'. But for bridge type interface connected to ovs bridge, it is different. "burst='2048'" will results in "burst 2000Kb" in tc outputs. <interface type='bridge'> <mac address='52:54:00:da:b9:fb'/> <source bridge='ovsbr0'/> <virtualport type='openvswitch'> <parameters interfaceid='67f7ce64-6ca9-4bb1-9309-f244a4e12213'/> </virtualport> <bandwidth> <inbound average='2000' peak='3000' burst='2048'/> <outbound average='256' peak='512' burst='512'/> </bandwidth> <target dev='vnet17'/> ... # tc class show dev vnet17 class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst **2000Kb** cburst 2000Kb class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b (In reply to yalzhang from comment #26) This is the way we document burst: from https://libvirt.org/formatnetwork.html: burst Optional attribute which specifies the amount of kilobytes that can be transmitted in a single burst at peak speed. > > # virsh domiflist rhel9 > Interface Type Source Model MAC > ----------------------------------------------------------------- > vnet15 network default virtio 52:54:00:dc:04:44 > > # virsh domiftune rhel9 vnet15 > inbound.average: 2000 > inbound.peak : 3000 > inbound.burst : 2048 > inbound.floor : 0 > outbound.average: 256 > outbound.peak : 512 > outbound.burst : 512 Therefore, burst=2048 means 2048 kilobytes = 2048000 bytes. > > # tc class show dev vnet15 > class htb 1:1 root leaf 2: prio 0 rate 16Mbit ceil 24Mbit burst **2Mb** > cburst 1596b > This is plain stupidity of how TC displays units. 2Mb in fact is 2*1024*1024 = 2097152 bytes. Consider the following example: # tc class add dev vnet2 parent 1:1 classid 1:2 htb rate 1000kbps burst 2097152 # tc class show dev vnet2 class htb 1:1 root rate 8192Kbit ceil 32768Kbit burst 2Mb cburst 1593b class htb 1:2 parent 1:1 prio 0 rate 8Mbit ceil 8Mbit burst 2Mb cburst 1600b Therefore, our documentation should be fixed to mention "kibibytes" instead of kilobytes. That's what we've always configured. > From the tc outputs, set "inbound.average: 2000" results in rate 16Mbit, the > unit is '1000'. > While for "inbound.burst : 2048", it results in "burst 2Mb", the unit is > '1024'. > Yes, that's correct. Size is in multiples of 1024 (just like everywhere else in libvirt) but rate is in multiples of 1000 (= standard SI multiples), because we are no longer talking about bytes, but individual bits. While confusing that's how other tools approach the units too - just like we saw with TC a few lines above. > But for bridge type interface connected to ovs bridge, it is different. > "burst='2048'" will results in "burst 2000Kb" in tc outputs. > <interface type='bridge'> > <mac address='52:54:00:da:b9:fb'/> > <source bridge='ovsbr0'/> > <virtualport type='openvswitch'> > <parameters interfaceid='67f7ce64-6ca9-4bb1-9309-f244a4e12213'/> > </virtualport> > <bandwidth> > <inbound average='2000' peak='3000' burst='2048'/> > <outbound average='256' peak='512' burst='512'/> > </bandwidth> > <target dev='vnet17'/> > ... > # tc class show dev vnet17 > class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst **2000Kb** > cburst 2000Kb > class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b Yeah, this should have been 2048Kb = 2Mb. Let me post a patch for that too. Patches posted here: https://listman.redhat.com/archives/libvir-list/2022-January/msg00094.html Merged upstream as: e429c062cb virnetdevopenvswitch: Fix 'burst' value passed to ovs-vsctl f3b422d9cc docs: Clarify 'burst' units for QoS v7.10.0-413-ge429c062cb For comment 29, test on libvirt-8.0.0-0rc1.1.el9.x86_64, the result is expected. # virsh dumpxml rh9 | grep /interface -B14 <interface type='bridge'> <mac address='52:54:00:da:b9:fb'/> <source bridge='ovsbr0'/> <virtualport type='openvswitch'> <parameters interfaceid='67f7ce64-6ca9-4bb1-9309-f244a4e12213'/> </virtualport> <bandwidth> <inbound average='2000' peak='3000' burst='2048'/> <outbound average='256' peak='512' burst='512'/> </bandwidth> <target dev='vnet58'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:53:be:14'/> <source network='default' portid='8b55c007-9d0a-49c9-82db-6b479ee5f415' bridge='virbr0'/> <bandwidth> <inbound average='2000' peak='3000' burst='2048'/> <outbound average='256' peak='512' burst='512'/> </bandwidth> <target dev='vnet59'/> <model type='virtio'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </interface> [root@sriov ~]# tc class show dev vnet59 class htb 1:1 root leaf 2: prio 0 rate 16Mbit ceil 24Mbit burst 2Mb cburst 1596b [root@sriov ~]# [root@sriov ~]# tc class show dev vnet58 class htb 1:1 parent 1:fffe prio 0 rate 16Mbit ceil 24Mbit burst 2Mb cburst 2Mb class htb 1:fffe root rate 24Mbit ceil 24Mbit burst 1497b cburst 1497b Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:1759 |