Bug 2105263

(In reply to arn from comment #0)
> Description of problem:
> 
> This is with regards to minQoS hw offload support on Mellanox NIC [CX-5 and
> 6]
> 
> Currently there is no way to set minQoS and have it offloaded apart from
> using IPlink / Devlink commands. But for ml2/ovn these commands cannot be
> used for reason below.
> "In ml2/ovn we have no idea on the VF rather we know the Representator port. 
> Furthermore even if we do know , to use iplink/devlink we need root access
> to set it and the OVN controller doesn't know how to do that and so it is
> not possible with the current design."
> 
> I tried doing changes in OVN to set QoS to the representator port. Due to
> changes, I can see it is setting to 'tc' .
> But it didn't have an effect on the transfer rates. Logs below.
> 
> Initially nothing is set.
> [root@rhos-nfv-09 ~]# tc qdisc show | grep htb
> [root@rhos-nfv-09 ~]#
> [root@rhos-nfv-09 ~]# tc class show dev enp4s0f0_0 | grep ceil
> [root@rhos-nfv-09 ~]#
> The VF is enp4s0f0v0 as below.. The corresponding representator is
> enp4s0f0_0  
> [root@rhos-nfv-09 ~]# ip a | grep enp4s0f0v0
> 46: enp4s0f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
> group default qlen 1000
>     inet 6.6.6.100/24 scope global enp4s0f0v0
> [root@rhos-nfv-09 ~]#
>  
> Now applying the qos. Note the rate is in bits/sec unit.
>  
> [root@rhos-nfv-09 ~]# ovn-nbctl set Logical_Switch_Port sw0-port1
> options:qos_max_rate=10000000
> [root@rhos-nfv-09 ~]# tc qdisc show | grep htb
> qdisc htb 1: dev enp4s0f0_0 root refcnt 561 r2q 10 default 0x1
> direct_packets_stat 0 direct_qlen 1000
> [root@rhos-nfv-09 ~]#
> [root@rhos-nfv-09 ~]#
> [root@rhos-nfv-09 ~]# tc class show dev enp4s0f0_0 | grep ceil
> class htb 1:fffe root rate 100Mbit ceil 100Mbit burst 1500b cburst 1500b
> class htb 1:2 parent 1:fffe prio 0 rate 12Kbit ceil 10Mbit burst 1563b
> cburst 1563b
> [root@rhos-nfv-09 ~]# 
>  
> We can see that qos has been applied on the representator port enp4s0f0_0 .
> Now when I tried running the iperf3 didn't notice much effect and it is the
> same behavior when no qos was applied.
>  
> [root@rhos-nfv-09 ~]# iperf3 -c 6.6.6.6
> Connecting to host 6.6.6.6, port 5201
> [  5] local 6.6.6.100 port 50972 connected to 6.6.6.6 port 5201
> [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> [  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec  982   1.00 MBytes      
> [  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec   18   1.25 MBytes      
> [  5]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec    5    988 KBytes      
> [  5]   3.00-4.00   sec  1.09 GBytes  9.38 Gbits/sec   17   1.23 MBytes      
> [  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec   12   1000 KBytes      
> [  5]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec    6    897 KBytes      
> [  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec   15   1.01 MBytes      
> [  5]   7.00-8.00   sec  1.09 GBytes  9.38 Gbits/sec   18    670 KBytes      
> [  5]   8.00-9.00   sec  1.09 GBytes  9.38 Gbits/sec    9   1.02 MBytes      
> [  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec   28    403 KBytes      
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bitrate         Retr
> [  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec  1110             sender
> [  5]   0.00-10.04  sec  10.9 GBytes  9.34 Gbits/sec                 
> receiver
>  
> iperf Done.
> [root@rhos-nfv-09 ~]# 
>  
> However there were some changes in the OVS flows though. I see the outgoing
> flow not offloaded and its setting skb_priority in the actions
>  
> [root@rhos-nfv-09 ovn]# ovs-appctl dpctl/dump-flows -m
> ufid:d20997a3-3515-4ec4-b2f6-5c7edb1547a3,
> skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),
> ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,
> id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=6a:62:68:b2:61:f6),eth_type(0x8100),
> vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.
> 0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no)), packets:209929,
> bytes:13859236, used:1.000s, offloaded:yes, dp:tc,
> actions:pop_vlan,enp4s0f0_0
> ufid:0425103c-e7c3-4fb0-8543-8d1531b180fd,
> recirc_id(0),dp_hash(0/0),skb_priority(0),in_port(enp4s0f0_0),skb_mark(0/0),
> ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=6a:62:68:b2:61:
> f6,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.
> 0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:77050,
> bytes:4926356768, used:0.000s, flags:SFP., dp:ovs,
> actions:push_vlan(vid=201,pcp=0),set(skb_priority(0x10002)),enp4s0f0
> [root@rhos-nfv-09 ovn]#
> 
> When I checked with NVIDIA understand that HTB support is there only for
> native mode i.e non-SRIOV.
> 
> This is an important use case from customer perspective and hence need to
> have HTB support extended to SRIOV as well. i.e even for switchdev mode when
> we set to the bandwidth share to representators it should take effect on VFs
> and we can support minQoS hw-offload well then.
> 
> Thanks & Regards,
> Abhiram R N
> 
> 
> Version-Release number of selected component (if applicable):
> [root@rhos-nfv-09 ~]# uname -r
> 4.18.0-305.22.1.el8_4.x86_64
> [root@rhos-nfv-09 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux release 8.4 (Ootpa)
> [root@rhos-nfv-09 ~]#
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1.
> 2.
> 3.
> 
> Actual results:
> 
> 
> Expected results:
> 
> 
> Additional info:

Hi Abhiram,

I noticed that HTB API doesn't really support minRate settings and basically just has the rate ceiling as an option.

Is the expectation that we will also expand the API part itself to support the min rate setting?

(In reply to Ariel Levkovich from comment #1)
> (In reply to arn from comment #0)
> > Description of problem:
> > 
> > This is with regards to minQoS hw offload support on Mellanox NIC [CX-5 and
> > 6]
> > 
> > Currently there is no way to set minQoS and have it offloaded apart from
> > using IPlink / Devlink commands. But for ml2/ovn these commands cannot be
> > used for reason below.
> > "In ml2/ovn we have no idea on the VF rather we know the Representator port. 
> > Furthermore even if we do know , to use iplink/devlink we need root access
> > to set it and the OVN controller doesn't know how to do that and so it is
> > not possible with the current design."
> > 
> > I tried doing changes in OVN to set QoS to the representator port. Due to
> > changes, I can see it is setting to 'tc' .
> > But it didn't have an effect on the transfer rates. Logs below.
> > 
> > Initially nothing is set.
> > [root@rhos-nfv-09 ~]# tc qdisc show | grep htb
> > [root@rhos-nfv-09 ~]#
> > [root@rhos-nfv-09 ~]# tc class show dev enp4s0f0_0 | grep ceil
> > [root@rhos-nfv-09 ~]#
> > The VF is enp4s0f0v0 as below.. The corresponding representator is
> > enp4s0f0_0  
> > [root@rhos-nfv-09 ~]# ip a | grep enp4s0f0v0
> > 46: enp4s0f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
> > group default qlen 1000
> >     inet 6.6.6.100/24 scope global enp4s0f0v0
> > [root@rhos-nfv-09 ~]#
> >  
> > Now applying the qos. Note the rate is in bits/sec unit.
> >  
> > [root@rhos-nfv-09 ~]# ovn-nbctl set Logical_Switch_Port sw0-port1
> > options:qos_max_rate=10000000
> > [root@rhos-nfv-09 ~]# tc qdisc show | grep htb
> > qdisc htb 1: dev enp4s0f0_0 root refcnt 561 r2q 10 default 0x1
> > direct_packets_stat 0 direct_qlen 1000
> > [root@rhos-nfv-09 ~]#
> > [root@rhos-nfv-09 ~]#
> > [root@rhos-nfv-09 ~]# tc class show dev enp4s0f0_0 | grep ceil
> > class htb 1:fffe root rate 100Mbit ceil 100Mbit burst 1500b cburst 1500b
> > class htb 1:2 parent 1:fffe prio 0 rate 12Kbit ceil 10Mbit burst 1563b
> > cburst 1563b
> > [root@rhos-nfv-09 ~]# 
> >  
> > We can see that qos has been applied on the representator port enp4s0f0_0 .
> > Now when I tried running the iperf3 didn't notice much effect and it is the
> > same behavior when no qos was applied.
> >  
> > [root@rhos-nfv-09 ~]# iperf3 -c 6.6.6.6
> > Connecting to host 6.6.6.6, port 5201
> > [  5] local 6.6.6.100 port 50972 connected to 6.6.6.6 port 5201
> > [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
> > [  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec  982   1.00 MBytes      
> > [  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec   18   1.25 MBytes      
> > [  5]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec    5    988 KBytes      
> > [  5]   3.00-4.00   sec  1.09 GBytes  9.38 Gbits/sec   17   1.23 MBytes      
> > [  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec   12   1000 KBytes      
> > [  5]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec    6    897 KBytes      
> > [  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec   15   1.01 MBytes      
> > [  5]   7.00-8.00   sec  1.09 GBytes  9.38 Gbits/sec   18    670 KBytes      
> > [  5]   8.00-9.00   sec  1.09 GBytes  9.38 Gbits/sec    9   1.02 MBytes      
> > [  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec   28    403 KBytes      
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval           Transfer     Bitrate         Retr
> > [  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec  1110             sender
> > [  5]   0.00-10.04  sec  10.9 GBytes  9.34 Gbits/sec                 
> > receiver
> >  
> > iperf Done.
> > [root@rhos-nfv-09 ~]# 
> >  
> > However there were some changes in the OVS flows though. I see the outgoing
> > flow not offloaded and its setting skb_priority in the actions
> >  
> > [root@rhos-nfv-09 ovn]# ovs-appctl dpctl/dump-flows -m
> > ufid:d20997a3-3515-4ec4-b2f6-5c7edb1547a3,
> > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),
> > ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0),packet_type(ns=0/0,
> > id=0/0),eth(src=e4:43:4b:4d:f1:10,dst=6a:62:68:b2:61:f6),eth_type(0x8100),
> > vlan(vid=201,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.
> > 0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no)), packets:209929,
> > bytes:13859236, used:1.000s, offloaded:yes, dp:tc,
> > actions:pop_vlan,enp4s0f0_0
> > ufid:0425103c-e7c3-4fb0-8543-8d1531b180fd,
> > recirc_id(0),dp_hash(0/0),skb_priority(0),in_port(enp4s0f0_0),skb_mark(0/0),
> > ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=6a:62:68:b2:61:
> > f6,dst=e4:43:4b:4d:f1:10),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.
> > 0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:77050,
> > bytes:4926356768, used:0.000s, flags:SFP., dp:ovs,
> > actions:push_vlan(vid=201,pcp=0),set(skb_priority(0x10002)),enp4s0f0
> > [root@rhos-nfv-09 ovn]#
> > 
> > When I checked with NVIDIA understand that HTB support is there only for
> > native mode i.e non-SRIOV.
> > 
> > This is an important use case from customer perspective and hence need to
> > have HTB support extended to SRIOV as well. i.e even for switchdev mode when
> > we set to the bandwidth share to representators it should take effect on VFs
> > and we can support minQoS hw-offload well then.
> > 
> > Thanks & Regards,
> > Abhiram R N
> > 
> > 
> > Version-Release number of selected component (if applicable):
> > [root@rhos-nfv-09 ~]# uname -r
> > 4.18.0-305.22.1.el8_4.x86_64
> > [root@rhos-nfv-09 ~]# cat /etc/redhat-release
> > Red Hat Enterprise Linux release 8.4 (Ootpa)
> > [root@rhos-nfv-09 ~]#
> > 
> > How reproducible:
> > 
> > 
> > Steps to Reproduce:
> > 1.
> > 2.
> > 3.
> > 
> > Actual results:
> > 
> > 
> > Expected results:
> > 
> > 
> > Additional info:
> 
> Hi Abhiram,
> 
> I noticed that HTB API doesn't really support minRate settings and basically
> just has the rate ceiling as an option.
> 
> Is the expectation that we will also expand the API part itself to support
> the min rate setting?

Hi Ariel,

From my understanding the rate and ceil options should suffice. 

From the OVS code what I saw is 
    tc_fill_rate(&opt.rate, class->min_rate, mtu);
    tc_fill_rate(&opt.ceil, class->max_rate, mtu);

And if I recollect correctly(its been a while)
I saw in driver code is they are converting this 'rate' (min_rate) what we have passed to bandwidth share.

So, basically the 'rate' is min_rate and 'ceil' is max_rate.

Thanks & Regards,
Abhiram R N

Ha, thanks Abhiram.

Guess the description is a bit confusing and saying max rate guaranteed. That's what threw me off I suppose.

Ha, thanks Abhiram.

Guess the description is a bit confusing and saying max rate guaranteed. That's what threw me off I suppose.

(In reply to arn from comment #2)
> From my understanding the rate and ceil options should suffice. 
> 
> From the OVS code what I saw is 
>     tc_fill_rate(&opt.rate, class->min_rate, mtu);
>     tc_fill_rate(&opt.ceil, class->max_rate, mtu);
> 
> And if I recollect correctly(its been a while)
> I saw in driver code is they are converting this 'rate' (min_rate) what we
> have passed to bandwidth share.
> 
> So, basically the 'rate' is min_rate and 'ceil' is max_rate.

Which are, from the htb man page https://www.man7.org/linux/man-pages/man8/tc-htb.8.html :

       rate rate
              Maximum rate this class and all its children are guaranteed. Mandatory.

       ceil rate
              Maximum rate at which a class can send, if its parent has bandwidth to spare.   De‐
              faults to the configured rate, which implies no borrowing

So "rate" can be understood as "the minimum bandwidth that it will have available". Quite confusing :)

> Currently there is no way to set minQoS and have it offloaded apart from using IPlink / Devlink commands. But for ml2/ovn these commands cannot be used for reason below.
> "In ml2/ovn we have no idea on the VF rather we know the Representator port. 

CMS could tell OVN controller which VF to use using LSP options, no?

> Furthermore even if we do know , to use iplink/devlink we need root access to set it and the OVN controller doesn't know how to do that and so it is not possible with the current design."

So the challenge is two-fold: 1) run devlink commands in context of ovn-controller; and 2) run the commands that presumably require root access, correct?

OVN supports so called vif provider interface: https://docs.ovn.org/en/stable/topics/vif-plug-providers/vif-plug-providers.html

This allows to run custom code for particularly tagged (typed) ports on ovn-controller side. See ovn-vif provider for example: https://github.com/ovn-org/ovn-vif/blob/main/lib/vif-plug-providers/representor/vif-plug-representor.c (This library is built-in in Ubuntu ovn packages btw, see here: https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/ovn/tree/ovn-vif.)

Note that the provider gathers information about VF/PF ports udev|devlink interface (see devlink_port_dump): https://github.com/ovn-org/ovn-vif/blob/b38abf036438c7354d307e8b8e3b08f92f6f38ce/lib/vif-plug-providers/representor/vif-plug-representor.c#L495

I don't know about whether set command indeed demands for root access. If so, couldn't this be handled through selinux attributes?

(In reply to Ihar Hrachyshka from comment #6)

Hi Ihar,

> > Currently there is no way to set minQoS and have it offloaded apart from using IPlink / Devlink commands. But for ml2/ovn these commands cannot be used for reason below.
> > "In ml2/ovn we have no idea on the VF rather we know the Representator port. 
> 
> CMS could tell OVN controller which VF to use using LSP options, no?
That is good I guess. We can use it to set on the VF instead of representator port.
> 
> > Furthermore even if we do know , to use iplink/devlink we need root access to set it and the OVN controller doesn't know how to do that and so it is not possible with the current design."
> 
> So the challenge is two-fold: 1) run devlink commands in context of
> ovn-controller; and 2) run the commands that presumably require root access,
> correct?
YES
> 
> OVN supports so called vif provider interface:
> https://docs.ovn.org/en/stable/topics/vif-plug-providers/vif-plug-providers.
> html
> 
> This allows to run custom code for particularly tagged (typed) ports on
> ovn-controller side. See ovn-vif provider for example:
> https://github.com/ovn-org/ovn-vif/blob/main/lib/vif-plug-providers/
> representor/vif-plug-representor.c (This library is built-in in Ubuntu ovn
> packages btw, see here:
> https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/ovn/tree/ovn-vif.
> )
> 
> Note that the provider gathers information about VF/PF ports udev|devlink
> interface (see devlink_port_dump):
> https://github.com/ovn-org/ovn-vif/blob/
> b38abf036438c7354d307e8b8e3b08f92f6f38ce/lib/vif-plug-providers/representor/
> vif-plug-representor.c#L495
> 
> I don't know about whether set command indeed demands for root access. If
> so, couldn't this be handled through selinux attributes?

I checked the VIF code but for them they dont need root access as they are using 
DEVLINK_CMD_PORT_GET and In the kernel code I can see below comment for that
/* can be retrieved by unprivileged users */

So we will need root access for the SET.

And from our internal discussions and with Marcelo and our team it was felt that a cleaner / good solution is to extend HTB support for SRIOV ( switdev mode I mean) as already min and max APIs are available.

Hi Amir/Ariel,

Do we have any update on this?

Just for clarity below 2 things mainly expected of this BZ

a) HTB support for SRIOV (switchdev mode) 
b) HTB hw-Offload

(Was not sure a) includes b)!. So, explicitly I have put that)

Thanks & Regards,
Abhiram R N

On the call yesterday Nvidia shared that there are some problems with this design.
Mainly, when we have this today:

(In reply to arn from comment #0)
> [root@rhos-nfv-09 ~]# tc class show dev enp4s0f0_0 | grep ceil
> class htb 1:fffe root rate 100Mbit ceil 100Mbit burst 1500b cburst 1500b
> class htb 1:2 parent 1:fffe prio 0 rate 12Kbit ceil 10Mbit burst 1563b
> cburst 1563b

On a VF representor, the VF representor directions are crossed with the regards of the traffic. That is, VFrep TX = Guest RX.
So if we install a HTB qdisc on VFrep egress, it should mean that it is about the Guest RX traffic.
That's why the filters used by OVS are installed on the ingress side of it.
And then, we can't really have HTB qdisc on ingress..
One idea would be for the driver "do the right thing under the hood", but that can lead to problems such as the sw datapath not respecting the QoS.

This problem doesn't exist with the uplink port, though.

I guess then one question here is: do we really need it to be per VF? Do we want to limit intra-host/inter-VF traffic as well or just control how much of the uplink each guest can use?

(In reply to Marcelo Ricardo Leitner from comment #9)
> On the call yesterday Nvidia shared that there are some problems with this
> design.
> Mainly, when we have this today:
> 
> (In reply to arn from comment #0)
> > [root@rhos-nfv-09 ~]# tc class show dev enp4s0f0_0 | grep ceil
> > class htb 1:fffe root rate 100Mbit ceil 100Mbit burst 1500b cburst 1500b
> > class htb 1:2 parent 1:fffe prio 0 rate 12Kbit ceil 10Mbit burst 1563b
> > cburst 1563b
> 
> On a VF representor, the VF representor directions are crossed with the
> regards of the traffic. That is, VFrep TX = Guest RX.
> So if we install a HTB qdisc on VFrep egress, it should mean that it is
> about the Guest RX traffic.
> That's why the filters used by OVS are installed on the ingress side of it.
> And then, we can't really have HTB qdisc on ingress..
> One idea would be for the driver "do the right thing under the hood", but
> that can lead to problems such as the sw datapath not respecting the QoS.
> 

Understood

> This problem doesn't exist with the uplink port, though.

Yes
> 
> I guess then one question here is: do we really need it to be per VF? Do we
> want to limit intra-host/inter-VF traffic as well or just control how much
> of the uplink each guest can use?

Let me tell you the use case, VF when consumed by VM, will be running certain traffic which has higher priorities over other VFs sharing same Uplink. While others can be Best effort, high priority VF should be minimum bandwidth gauranteed from the uplink port bandwidth.

So, If uplink port is able to reserve the bandwidth for such VFs and limit others, Having HTB qdisc on the uplink should be fine. Regarding Intra VF, I think with Legacy SRIOV as well, we dont have Intra VF bandwidth guaranteed. Intra VF in Legacy too is Best effort. Thus, our case here is Inter VF traffic. 

Does it make sense?

Thanks

(In reply to Haresh Khandelwal from comment #10)
> Does it make sense?

It does. Thanks.

And yes, htb just on uplink is able to solve it as we can have openflow flows steering traffic from specific ports to specific queues (bandwidth reserves).
Today OVN, tough, is mangling is directly, bypassing OVS, but we likely can improve that.

It was decided to not base this feature on HTB HWOL as it wasn't possible to find a good model that works nicely both in sw and in hw.

For the actual implementation then, please refer to OSP team.

Thanks!