Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1761407

Summary: [RHEL 8] QoS Max bandwidth limit doesn't work in combination with DVR
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: ovn2.11Assignee: lorenzo bianconi <lorenzo.bianconi>
Status: CLOSED CURRENTRELEASE QA Contact: haidong li <haili>
Severity: medium Docs Contact:
Priority: medium    
Version: FDP 19.GCC: amuller, apevec, chrisw, ctrautma, dalvarez, dcbw, fhallal, haili, lorenzo.bianconi, majopela, mmichels, nusiddiq, ovs-qe, qding, rhos-maint, srevivo, yinxu
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1580542 Environment:
Last Closed: 2020-11-12 16:44:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1580542, 1851493    
Bug Blocks: 1638040, 1640041, 1704272    

Description Numan Siddique 2019-10-14 10:57:56 UTC
+++ This bug was initially created as a clone of Bug #1580542 +++

Description of problem:

   This is the U/S version of the bug:
    
      https://github.com/openvswitch/ovs-issues/issues/150

   When a port has assigned qos_max_rate  / qos_burst and at the same time the port is directly attached to a provider network logical switch (with a physical mapping), or it has a distributed floating IP (which won't go through a tunnel to a gateway chassis), the bandwidth limit has no effect since the queues are only configured to tunnel ports, and not also on physical interfaces of the bridge mappings.


Version-Release number of selected component (if applicable):

2.9.0

How reproducible:

Always

--- Additional comment from Miguel Angel Ajo on 2018-05-21 18:25:55 UTC ---


DescriptionEdit
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-networking-ovn-13_director-rhel-virthost-3cont_2comp-ipv4-geneve-dvr/lastCompletedBuild/testReport/neutron_tempest_plugin.scenario.test_qos/QoSTest/test_qos_id_1f7ed39b_428f_410a_bd2b_db9f465680df_/

--- Additional comment from Mark Michelson on 2018-06-27 18:35:00 UTC ---

Hi Miguel,

I've looked at this a bit, and I think I have a better understanding of what you're saying is the issue.

OVN currently will only allocate QOS queues if it can determine a tunnel interface on which to assign the queue. It does this by looking at interfaces on br-int, and seeing which of those has options:remote_ip set. It then puts the QOS queue on that interface's status:tunnel_egress_interface.

In your case, you have non-tunnel interfaces on which you want to enable QOS queues. So the question here is how OVN can determine that it should add a QOS queue to an interface.

The most explicit way to do it would be to place something in the Interface's options or external_ids, like "qos_egress_interface", and then apply the QOS queue to that. There may be a more "automatic" way of doing this though, such as using the Interface's "type" as a way to determine if the QOS queue should be applied.

Do you have any ideas in mind that would work here? I'm not familiar with what OpenStack's OVN/OVS configuration looks like in this particular case, so seeing what you have could provide some good hints.

--- Additional comment from Scott Lewis on 2018-07-26 16:22:15 UTC ---

Removing TM until this is targeted to a specific release.

--- Additional comment from Jakub Sitnicki on 2018-07-27 16:56:03 UTC ---

Mark,

I belive we now have a better understanding of how OpenStack is using
"localnet" ports after the discussions on MAC flapping and the
explanation in Miguel's post:

https://ajo.es/ovn-distributed-ew-on-vlan/

This is the same setup that Miguel's refers to in this BZ. As you
pointed out, ovn-controller only installs qdisc's on OVN tunnel egress
devices, so it won't work.

What bothers me most, though, is why we don't simply configure qdisc's
on the VIF (be it tap, veth, or OVS internal interface)? This is a
documented solution:

https://github.com/openvswitch/ovs/blob/v2.9.2/Documentation/howto/qos.rst

Is it because we want to limit just the traffic leaving the node? It
seems worth pursuing further to reach an explanation.

Otherwise, ovn-controller logic will have to be extended to find all
egress devices reachable from br-int. Might be tricky.

--- Additional comment from Mark Michelson on 2018-08-09 18:14:57 UTC ---

Hi Miguel,

We could use some guidance on how best to proceed with this. As Jakub pointed out, you have the option of setting up QoS ingress policing based on the link he posted above.

If you want egress QoS similar to what is done on tunnel ports, that requires some changes in OVN. In that case, understanding what would be acceptable from OpenStack's perspective would be useful. As Jakub mentioned, it would be tricky to find all egress devices reachable from br-int, so I would prefer not to go that route if possible.

I previously suggested that in the OpenVswitch Interface table, you could specify "external_ids:qos_egress_iface=<some device>" on the interface that is bound to the localnet logical switch port. I don't think this exact idea will work, because the localnet port will actually be bound to an automatically-created patch port rather than a port you add to br-int.

I think a potential answer might be to follow the patch port to br-phys (or br-ex, or whatever we're calling it), and place a QoS queue on interfaces on that bridge that have something like "external_ids:ovn_qos=yes" set and that are of type "system". This way, we satisfy the need to automatically add egress QoS queues, and we also have confirmation through configuration that the QoS queue is desired on the specific interface.

What do you think about this suggestion? If you don't like it, what would you suggest instead?

--- Additional comment from Miguel Angel Ajo on 2018-09-03 12:25:11 UTC ---

The suggestion seems reasonable to me.

I suspect ovn-controller is already (partially doing that), I discovered when I found the bug, that the egress queue was set to one interface on "br-`ex`", but since sometimes we deploy with several interfaces there (for other purposes), the right one didn't get it.

If that's the case -I wasn't able to follow the code properly- may be we will need to setup a list of interfaces like : external_ids:qos_egress_ifaces=<some device>,<another-device>,..."  ?

--- Additional comment from Mark Michelson on 2018-09-05 20:26:44 UTC ---

I'll have to take another look through the code to see if I can trace where that might be happening. From what I could tell, the only way egress qos qdiscs were set up was for tunnels. It may be that somehow an egress tunnel iface is being set up incorrectly by OVS? I'm not sure.

Your suggestion could work, too. It might be more efficient than my idea as well. I'll provide a patch for you to test and you can tell me if it works.

--- Additional comment from Mark Michelson on 2018-09-25 21:15:00 UTC ---

Hi Miguel. I've created a branch in my fork of OVS where I've attempted what I described earlier. I would appreciate if you could give this a test in your setup to see if it works as expected.

https://github.com/putnopvut/ovs/tree/qos-thing

The method I ended up going with is similar to what I initially discussed, but I decided to make it slightly more generic. On br-ex, you can set 'external-ids:ovn-egress-iface=true' on each interface that should have QoS qdiscs enabled. I have not tested this myself yet, so your feedback will be valuable.

--- Additional comment from Mark Michelson on 2018-11-13 21:45:52 UTC ---

Here's a scratch build of OVN 2.9 I made with the patch applied. Hopefully one of the links below will be useful for testing.

To reiterate how this is intended to work, in the bridge that br-int is connected to, you can specify on the egress interface "external-ids:ovn-egress-interface=true" and we should set up QoS on that interface. Once again, this has not been tested yet because I am not sure about the proper environment to test it in. So this may not work as expected.

A yum repository for the scratch build of openvswitch-2.9.0-81.el7fdn.bz1580542.1 (task 19171354) is available at:

http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/openvswitch-2.9.0-81.el7fdn.bz1580542.1-scratch.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm
http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm

--- Additional comment from errata-xmlrpc on 2019-10-11 15:41:22 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2019:47208-01
https://errata.devel.redhat.com/advisory/47208

Comment 1 haidong li 2019-10-17 09:23:15 UTC
Hi Lorenzo,

   Can you help describe the configuration of this feature? I have configured the commands below,but the qos didn't work.I configured the qos max rate on localnet logical port,and configure the external-ids on the patch-port of the br-ex.Is there any incorrect configuration?Thanks!

   ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=10000000
   ovs-vsctl set interface enp6s0f1 external-ids:ovn-egress-interface=true

Comment 2 haidong li 2019-10-30 07:37:25 UTC
This issue is verified on the latest version:
[root@dell-per730-19 ovn]# uname -a
Linux dell-per730-19.rhts.eng.pek2.redhat.com 4.18.0-147.6.el8.x86_64 #1 SMP Wed Oct 9 22:56:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per730-19 ovn]# rpm -qa | grep openvswitch
kernel-kernel-networking-openvswitch-ovs_qinq-1.3-34.noarch
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
openvswitch2.11-2.11.0-24.el8fdp.x86_64
[root@dell-per730-19 ovn]# rpm -qa | grep ovn
ovn2.11-2.11.1-8.el8fdp.x86_64
ovn2.11-host-2.11.1-8.el8fdp.x86_64
ovn2.11-central-2.11.1-8.el8fdp.x86_64
[root@dell-per730-19 ovn]# ovn-nbctl show
switch 0bae869d-cbbf-4eca-9636-eb21131a8b46 (public)
    port public_r1
        type: router
        router-port: r1_public
    port ln_p1
        type: localnet
        addresses: ["unknown"]
switch 900296f2-b168-4776-91b9-63962c3b1cec (s2)
    port hv1_vm00_vnet1
        addresses: ["00:de:ad:01:00:01 172.16.102.11"]
    port hv1_vm01_vnet1
        addresses: ["00:de:ad:01:01:01 172.16.102.12"]
    port s2_r1
        type: router
        addresses: ["00:de:ad:ff:01:02 172.16.102.1"]
        router-port: r1_s2
switch 86cbc7c3-bab1-47db-abe2-032775927c2a (s3)
    port s3_r1
        type: router
        addresses: ["00:de:ad:ff:01:03 172.16.103.1"]
        router-port: r1_s3
    port hv0_vm00_vnet1
        addresses: ["00:de:ad:00:00:01 172.16.103.11"]
    port hv0_vm01_vnet1
        addresses: ["00:de:ad:00:01:01 172.16.103.12"]
router 34ce4801-21ad-4c1d-b8bc-bd8e8f27d3b2 (r1)
    port r1_s3
        mac: "00:de:ad:ff:01:03"
        networks: ["172.16.103.1/24"]
    port r1_public
        mac: "40:44:00:00:00:03"
        networks: ["172.16.104.1/24"]
    port r1_s2
        mac: "00:de:ad:ff:01:02"
        networks: ["172.16.102.1/24"]
    nat 09ed872a-33e3-47c4-9c10-d67b9f71a0b6
        external ip: "172.16.104.201"
        logical ip: "172.16.103.11"
        type: "dnat_and_snat"
    nat e40bea83-7874-4066-a804-6ee2771d3baa
        external ip: "172.16.104.200"
        logical ip: "172.16.102.11"
        type: "dnat_and_snat"
[root@dell-per730-19 ovn]# 
[root@dell-per730-19 ovn]#  ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=1000
[root@dell-per730-19 ovn]#    ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_burst=1000
[root@dell-per730-19 ovn]# ovs-vsctl set interface enp131s0f1 external-ids:ovn-egress-iface=true
[root@dell-per730-19 ovn]# virsh console hv1_vm00
Connected to domain hv1_vm00
Escape character is ^]

[root@localhost ~]# netperf -H 172.16.104.201
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.104.201 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    14.00       0.02   
[root@localhost ~]#