Bug 2002406

If I unset the neutron qos policy and leave VF with max qos 0 (disabled)

openstack port unset --qos-policy vlanport2

vf 1     link/ether f8:f2:1e:03:bf:f3 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off

[root@vm2 ~]# iperf3 -c 6.6.6.6  <<<<<<<<<<<<I could achieve ~9.39 gbps
Connecting to host 6.6.6.6, port 5201
[  4] local 6.6.6.102 port 35472 connected to 6.6.6.6 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec   49    915 KBytes       
[  4]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec    7    963 KBytes       
[  4]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec    9   1014 KBytes       
[  4]   3.00-4.00   sec  1.09 GBytes  9.38 Gbits/sec   21   1.01 MBytes       
[  4]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec   29    848 KBytes       
[  4]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec   21    911 KBytes       
[  4]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec   10    963 KBytes       
[  4]   7.00-8.00   sec  1.09 GBytes  9.38 Gbits/sec    9   1020 KBytes       
[  4]   8.00-9.00   sec  1.09 GBytes  9.38 Gbits/sec   35   1.03 MBytes       
[  4]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec   10   1.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec  200             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver

iperf Done.

As flows are offlaoded. 

ufid:1fc13947-5277-4950-a931-9b2c565a803a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(enp4s0f0_1),packet_type(ns=0/0,id=0/0),eth(src=f8:f2:1e:03:bf:f3,dst=c2:64:5c:2b:8f:75),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=6.6.6.4/255.255.255.252,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:7815200, bytes:11863449200, used:0.610s, offloaded:yes, dp:tc, actions:push_vlan(vid=415,pcp=0),mx-bond

ufid:2d65df8b-0c67-467d-8f98-3a57ed71b7da, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(mx-bond),packet_type(ns=0/0,id=0/0),eth(src=c2:64:5c:2b:8f:75,dst=f8:f2:1e:03:bf:f3),eth_type(0x8100),vlan(vid=415,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no)), packets:242339, bytes:15749194, used:0.610s, offloaded:yes, dp:tc, actions:pop_vlan,enp4s0f0_1

Hi, Haresh.

Offloading of meters is not supported by OVS.  Though, there is an ongoing
work in this direction, but it's in early stage:
  https://patchwork.ozlabs.org/project/openvswitch/list/?series=260741&state=*
Even if this patch set will be accepted, functionality will not be available
until OVS 2.17 release in February.

For the inconsistency of kernel meters, I think, at least we need to backport
following patches from the upstream kernel, if they are not already:
  e4df1b0c2435 ("openvswitch: meter: fix race when getting now_ms.")
  7d742b509dd7 ("openvswitch: meter: remove rate from the bucket size calculation")

This should fix some problems with inadequate rates.

Marcelo, what do you think?

one more thing, we need those kernel patches as well to improve rate limit mechanism.
I did small test, Have a VF and applied 2 gbps max qos rate limit, now sending 2 streams traffic, so overall traffic rate limit for this VF remains 2 gbps.

For Max qos applied directly on VF, it is 905 Mbits/sec for 1st stream and 1.005 Gbits/sec for 2nd stream. On Average, this comes ~2 gbps.
For Max qos applied via metering flows,   it is 977 Mbits/sec for 1st stream and 844 Mbit/sec for 2nd stream. On average, this comes ~1.821 gbps.

This is filed under an OVN component currently. Should this be moved to OVS instead?

Oh. Yes.

I suppose I can take this bz for now while we understand all the dependencies it may have (ovs, kernel tc, driver).
There are patches posted upstream already that could use review from OVS team as well, though.

[PATCH v2 0/6] Add support for ovs metering with tc offload

I'll leave it up to the OVS team to reassign to me or keep it.

Oops, wrong button ;-) reassigning it back.

List of patches in U/S: https://patchwork.ozlabs.org/project/openvswitch/list/?submitter=80786

This effort is driven by Nvidia, should we assign this BZ to them?

Hi Amir. Parking this one with you until the patchset is accepted upstream. Then we'll take it back and handle the downstream portion. Thanks.

vswitchd patches: https://patchwork.ozlabs.org/project/openvswitch/cover/20220708095533.32489-1-jianbol@nvidia.com/
mlx5 driver patches: https://lore.kernel.org/netdev/20220702190213.80858-1-saeed%40kernel.org/T/
flow_offload patches: https://lore.kernel.org/netdev/20220224102908.5255-1-jianbol%40nvidia.com/T/

My understanding then is that this is done upstream.
Amir, can you please confirm?

We confirmed on the call today that this is considered a done feature by Nvidia.

So this is part of OVS 3.0 and will not be backported. So from an OVS perspective, this is done. 

Marcelo, I guess we can close this BZ, and a new (clone) can be opened for the kernel part.

(In reply to Marcelo Ricardo Leitner from comment #18)
> vswitchd patches:
> https://patchwork.ozlabs.org/project/openvswitch/cover/20220708095533.32489-
> 1-jianbol/

In OVS v3.0 per comment above.

> mlx5 driver patches:
> https://lore.kernel.org/netdev/20220702190213.80858-1-saeed%40kernel.org/T/

Landed in v6.0 upstream.
targeted for 9.2: https://bugzilla.redhat.com/show_bug.cgi?id=2049629#c2
targeted for 8.8: https://bugzilla.redhat.com/show_bug.cgi?id=2049622#c5

> flow_offload patches:
> https://lore.kernel.org/netdev/20220224102908.5255-1-jianbol%40nvidia.com/T/

in 9.2 via https://bugzilla.redhat.com/show_bug.cgi?id=2128185
in 8.7 via https://bugzilla.redhat.com/show_bug.cgi?id=2106271


Haresh, how do these versions work for OSP?

(In reply to Eelco Chaudron from comment #20)
> So this is part of OVS 3.0 and will not be backported. So from an OVS
> perspective, this is done. 
> 
> Marcelo, I guess we can close this BZ, and a new (clone) can be opened for
> the kernel part.

Perhaps by now we can convert this bz into a TestOnly one, so that we're sure we have a test at OVS level for it.
There is a test case shared by Amir on the driver rebases but I can't tell if QE would be using the right FDP OVS.
Jianlin, thoughts?

(In reply to Marcelo Ricardo Leitner from comment #22)
> (In reply to Eelco Chaudron from comment #20)
> > So this is part of OVS 3.0 and will not be backported. So from an OVS
> > perspective, this is done. 
> > 
> > Marcelo, I guess we can close this BZ, and a new (clone) can be opened for
> > the kernel part.
> 
> Perhaps by now we can convert this bz into a TestOnly one, so that we're
> sure we have a test at OVS level for it.
> There is a test case shared by Amir on the driver rebases but I can't tell
> if QE would be using the right FDP OVS.
> Jianlin, thoughts?

qijun, would ovs team help to add case to cover this feature?

(In reply to Jianlin Shi from comment #23)
> 
> qijun, would ovs team help to add case to cover this feature?

When the feature is available to test I will add a test case to cover the feature. Thank you.

(In reply to Marcelo Ricardo Leitner from comment #21)
> (In reply to Marcelo Ricardo Leitner from comment #18)
> > vswitchd patches:
> > https://patchwork.ozlabs.org/project/openvswitch/cover/20220708095533.32489-
> > 1-jianbol/
> 
> In OVS v3.0 per comment above.
> 
> > mlx5 driver patches:
> > https://lore.kernel.org/netdev/20220702190213.80858-1-saeed%40kernel.org/T/
> 
> Landed in v6.0 upstream.
> targeted for 9.2: https://bugzilla.redhat.com/show_bug.cgi?id=2049629#c2
> targeted for 8.8: https://bugzilla.redhat.com/show_bug.cgi?id=2049622#c5
> 
> > flow_offload patches:
> > https://lore.kernel.org/netdev/20220224102908.5255-1-jianbol%40nvidia.com/T/
> 
> in 9.2 via https://bugzilla.redhat.com/show_bug.cgi?id=2128185
> in 8.7 via https://bugzilla.redhat.com/show_bug.cgi?id=2106271
> 
> 
> Haresh, how do these versions work for OSP?

OSP17.1 will have fixed ovs and kernel then.

Thanks

Making this bz a TestOnly one per above.

Eelco, parking this with you.
This just needs to be verified.
fbl

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openvswitch3.1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:4685

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days