Bug 2060310

Summary: [RFE] Implement QoS minimum bandwidth rules
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Rodolfo Alonso <ralonsoh>
Component: ovn22.03Assignee: Ihar Hrachyshka <ihrachys>
Status: CLOSED ERRATA QA Contact: ying xu <yinxu>
Severity: medium Docs Contact:
Priority: medium    
Version: FDP 22.ECC: ctrautma, dalvarez, hakhande, ihrachys, jiji, ksundara, mheib, mmichels, shuali
Target Milestone: ---Keywords: FutureFeature, RFE
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovn22.03-22.03.0-74.el8fdp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2077681 (view as bug list) Environment:
Last Closed: 2022-10-11 12:21:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2077681    

Description Rodolfo Alonso 2022-03-03 09:41:03 UTC 
The goal of this RFE is to add QoS minimum bandwidth guarantee rule support in core OVN. This new functionality should close the parity gap between Neutron ML2/OVS and ML2/OVN.

The scope of this RFE is limited to ports connected to physical networks (flat, VLAN); that means, VM ports that will egress traffic through a physical bridge.


Neutron ML2/OVS
===============
Now in ML2/OVS we provide minimum bandwidth only for those ports connected to physical networks (flat, VLAN). The QoS min-BW rules for a VM port are set in the physical bridge port where the VM port traffic leaves the node. The rationale for this is that the min-BW rules pretend to guarantee a certain BW for the VM traffic where the traffic is aggregated; this is in the physical port interface. [1] is the ML2/OVS implementation.

In ML2/OVS, the physical bridge interface port has a QoS register associated. This QoS register has several Queue registers, one per VM port. Each Queue register has a specific "queue-num" associated and the corresponding "other-config:min-rate" value.


Proposal
========
This RFE proposes to add a new parameter in DB NB "QoS.bandwdth" enumerate: ["rate", "burst", "minimum"]. The "match", "direction" and "priority" columns should work as now. In order to identify the egress port (physical bridge port), this RFE proposed to add a reference in "external_ids" (this is just an idea that could wrong due to my lack of knowledge in OVN internals).


Please, for more information you can reach me in IRC (ralonsoh), mail and this BZ.

Regards and thank you in advance.

[1]https://github.com/openstack/neutron/blob/b072cbf05f079519e379b2bfebe27846ae612275/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py#L179-L190
Comment 1 Daniel Alvarez Sanchez 2022-03-03 15:07:21 UTC 
One important note is that, at the moment OVN is implementing QoS via netdev [0] and not using OVS meters when the parameters are applied to the LSP.options column [1] (instead of to the QoS table).

If we want HW offload, since OVS meters are not yet supported we could take a similar approach using either netdev/iproute to allow for HW offload?


[0] https://github.com/ovn-org/ovn/blob/5f7a985de0707539e4df1eab4c4150b24dbce622/controller/binding.c#L221
[1] https://github.com/ovn-org/ovn/blob/b988d5fa62d9dbecacb2c45defab8e7edde4c533/ovn-nb.xml#L1031-L1039
Comment 2 Ihar Hrachyshka 2022-04-20 12:20:35 UTC 
(In reply to Daniel Alvarez Sanchez from comment #1)
> One important note is that, at the moment OVN is implementing QoS via netdev
> [0] and not using OVS meters when the parameters are applied to the
> LSP.options column [1] (instead of to the QoS table).
> 
> If we want HW offload, since OVS meters are not yet supported we could take
> a similar approach using either netdev/iproute to allow for HW offload?
> 
> 
> [0]
> https://github.com/ovn-org/ovn/blob/5f7a985de0707539e4df1eab4c4150b24dbce622/
> controller/binding.c#L221
> [1]
> https://github.com/ovn-org/ovn/blob/b988d5fa62d9dbecacb2c45defab8e7edde4c533/
> ovn-nb.xml#L1031-L1039

This series should give us hw offload for ovs metering: https://patchwork.ozlabs.org/project/openvswitch/list/?series=293970
Comment 3 Ihar Hrachyshka 2022-04-27 15:31:24 UTC 
Minimalistic / backportable / no OVS metering implementation: https://patchwork.ozlabs.org/project/ovn/list/?series=297048
Comment 4 Ihar Hrachyshka 2022-05-31 22:11:03 UTC 
This is going to be included in upstream 22.06 release.
Comment 8 OVN Bot 2022-08-04 12:38:19 UTC 
ovn22.03 fast-datapath-rhel-9 clone created at https://bugzilla.redhat.com/show_bug.cgi?id=2115331
Comment 11 ying xu 2022-09-22 03:27:18 UTC 
# rpm -qa|grep ovn
ovn22.03-central-22.03.0-95.el8fdp.x86_64
ovn22.03-host-22.03.0-95.el8fdp.x86_64
ovn22.03-22.03.0-95.el8fdp.x86_64


# ovn-nbctl show
switch 4e657969-0058-4761-943a-477fee48981c (s3)
    port hv0_vm00_vnet1
        addresses: ["00:de:ad:00:00:01 172.16.103.11"]
    port hv0_vm01_vnet1
        addresses: ["00:de:ad:00:01:01 172.16.103.12"]
    port s3_r1
        type: router
        addresses: ["00:de:ad:ff:01:03 172.16.103.1"]
        router-port: r1_s3
switch b69e3e01-6850-497b-8c73-b71cc9397b85 (public)
    port public_r1
        type: router
        router-port: r1_public
    port ln_p1
        type: localnet
        addresses: ["unknown"]
switch 16343f82-4ea6-4fc5-8dcd-564b098fba29 (s2)
    port hv1_vm00_vnet1
        addresses: ["00:de:ad:01:00:01 172.16.102.11"]
    port hv1_vm01_vnet1
        addresses: ["00:de:ad:01:01:01 172.16.102.12"]
    port s2_r1
        type: router
        addresses: ["00:de:ad:ff:01:02 172.16.102.1"]
        router-port: r1_s2
router e812e570-59d1-4260-b6a0-04738fb7e3d3 (r1)
    port r1_s2
        mac: "00:de:ad:ff:01:02"
        networks: ["172.16.102.1/24"]
    port r1_s3
        mac: "00:de:ad:ff:01:03"
        networks: ["172.16.103.1/24"]
    port r1_public
        mac: "40:44:00:00:00:03"
        networks: ["172.16.104.1/24"]
        gateway chassis: [hv1]
    nat 299e7e7e-f916-447e-9d05-32d2a6ad99a0
        external ip: "172.16.104.201"
        logical ip: "172.16.103.11"
        type: "dnat_and_snat"
    nat 6d45b21e-f5c5-4ec4-b3ec-2dc3575acb91
        external ip: "172.16.104.200"
        logical ip: "172.16.102.11"
        type: "dnat_and_snat"


:: [ 23:16:59 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=2000000'
:: [ 23:16:59 ] :: [   PASS   ] :: Command 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=2000000' (Expected 0, got 0)
:: [ 23:16:59 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_min_rate=1000000'
:: [ 23:16:59 ] :: [   PASS   ] :: Command 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_min_rate=1000000' (Expected 0, got 0)
:: [ 23:16:59 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_burst=2200000'
:: [ 23:16:59 ] :: [   PASS   ] :: Command 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_burst=2200000' (Expected 0, got 0)
:: [ 23:16:59 ] :: [  BEGIN   ] :: Running 'tc qdisc show | grep htb'
qdisc htb 1: dev ens3f0 root refcnt 65 r2q 10 default 0x1 direct_packets_stat 2 direct_qlen 1000
qdisc htb 1: dev ens3f1 root refcnt 65 r2q 10 default 0x1 direct_packets_stat 0 direct_qlen 1000
:: [ 23:16:59 ] :: [   PASS   ] :: Command 'tc qdisc show | grep htb' (Expected 0, got 0)
:: [ 23:16:59 ] :: [  BEGIN   ] :: Running 'tc class show dev ens3f1'
class htb 1:fffe root rate 10Gbit ceil 10Gbit burst 1250b cburst 1250b 
class htb 1:2 parent 1:fffe prio 0 rate 1Mbit ceil 2Mbit burst 275000b cburst 275000b ---------------------------set success
:: [ 23:16:59 ] :: [   PASS   ] :: Command 'tc class show dev ens3f1' (Expected 0, got 0)

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.104.201 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    14.00       2.06   --------------traffic shape

change qos set
:: [ 23:17:26 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=20000000'
:: [ 23:17:26 ] :: [   PASS   ] :: Command 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=20000000' (Expected 0, got 0)
:: [ 23:17:26 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_burst=22000000'
:: [ 23:17:26 ] :: [   PASS   ] :: Command 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_burst=22000000' (Expected 0, got 0)
:: [ 23:17:26 ] :: [  BEGIN   ] :: Running 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_min_rate=10000000'
:: [ 23:17:26 ] :: [   PASS   ] :: Command 'ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_min_rate=10000000' (Expected 0, got 0)
:: [ 23:17:26 ] :: [  BEGIN   ] :: Running 'tc qdisc show | grep htb'
qdisc htb 1: dev ens3f0 root refcnt 65 r2q 10 default 0x1 direct_packets_stat 15 direct_qlen 1000
qdisc htb 1: dev ens3f1 root refcnt 65 r2q 10 default 0x1 direct_packets_stat 7 direct_qlen 1000
:: [ 23:17:26 ] :: [   PASS   ] :: Command 'tc qdisc show | grep htb' (Expected 0, got 0)
:: [ 23:17:27 ] :: [  BEGIN   ] :: Running 'tc class show dev ens3f1'
class htb 1:fffe root rate 10Gbit ceil 10Gbit burst 1250b cburst 1250b 
class htb 1:2 parent 1:fffe prio 0 rate 10Mbit ceil 20Mbit burst 2750000b cburst 2750000b -------------------change right
:: [ 23:17:27 ] :: [   PASS   ] :: Command 'tc class show dev ens3f1' (Expected 0, got 0)

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.104.201 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    11.16      21.01   -------------traffic shape
Comment 13 errata-xmlrpc 2022-10-11 12:21:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn22.03 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6870