Bug 1580542
| Summary: | [RHEL 7] QoS Max bandwidth limit doesn't work in combination with DVR | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Miguel Angel Ajo <majopela> | |
| Component: | ovn2.11 | Assignee: | lorenzo bianconi <lorenzo.bianconi> | |
| Status: | CLOSED ERRATA | QA Contact: | haidong li <haili> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | FDP 19.G | CC: | amuller, apevec, chrisw, ctrautma, dalvarez, dcbw, fhallal, haili, kfida, lorenzo.bianconi, majopela, mjozefcz, mmichels, nusiddiq, qding, rhos-maint, srevivo, yinxu | |
| Target Milestone: | --- | Keywords: | Triaged, ZStream | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | ovn2.11-2.11.1-4.el7fdn | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1761407 1778036 1851493 (view as bug list) | Environment: | ||
| Last Closed: | 2019-11-06 05:00:08 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1638040, 1640041, 1704272, 1761407, 1851493 | |||
Hi Miguel, I've looked at this a bit, and I think I have a better understanding of what you're saying is the issue. OVN currently will only allocate QOS queues if it can determine a tunnel interface on which to assign the queue. It does this by looking at interfaces on br-int, and seeing which of those has options:remote_ip set. It then puts the QOS queue on that interface's status:tunnel_egress_interface. In your case, you have non-tunnel interfaces on which you want to enable QOS queues. So the question here is how OVN can determine that it should add a QOS queue to an interface. The most explicit way to do it would be to place something in the Interface's options or external_ids, like "qos_egress_interface", and then apply the QOS queue to that. There may be a more "automatic" way of doing this though, such as using the Interface's "type" as a way to determine if the QOS queue should be applied. Do you have any ideas in mind that would work here? I'm not familiar with what OpenStack's OVN/OVS configuration looks like in this particular case, so seeing what you have could provide some good hints. Mark, I belive we now have a better understanding of how OpenStack is using "localnet" ports after the discussions on MAC flapping and the explanation in Miguel's post: https://ajo.es/ovn-distributed-ew-on-vlan/ This is the same setup that Miguel's refers to in this BZ. As you pointed out, ovn-controller only installs qdisc's on OVN tunnel egress devices, so it won't work. What bothers me most, though, is why we don't simply configure qdisc's on the VIF (be it tap, veth, or OVS internal interface)? This is a documented solution: https://github.com/openvswitch/ovs/blob/v2.9.2/Documentation/howto/qos.rst Is it because we want to limit just the traffic leaving the node? It seems worth pursuing further to reach an explanation. Otherwise, ovn-controller logic will have to be extended to find all egress devices reachable from br-int. Might be tricky. Hi Miguel, We could use some guidance on how best to proceed with this. As Jakub pointed out, you have the option of setting up QoS ingress policing based on the link he posted above. If you want egress QoS similar to what is done on tunnel ports, that requires some changes in OVN. In that case, understanding what would be acceptable from OpenStack's perspective would be useful. As Jakub mentioned, it would be tricky to find all egress devices reachable from br-int, so I would prefer not to go that route if possible. I previously suggested that in the OpenVswitch Interface table, you could specify "external_ids:qos_egress_iface=<some device>" on the interface that is bound to the localnet logical switch port. I don't think this exact idea will work, because the localnet port will actually be bound to an automatically-created patch port rather than a port you add to br-int. I think a potential answer might be to follow the patch port to br-phys (or br-ex, or whatever we're calling it), and place a QoS queue on interfaces on that bridge that have something like "external_ids:ovn_qos=yes" set and that are of type "system". This way, we satisfy the need to automatically add egress QoS queues, and we also have confirmation through configuration that the QoS queue is desired on the specific interface. What do you think about this suggestion? If you don't like it, what would you suggest instead? The suggestion seems reasonable to me. I suspect ovn-controller is already (partially doing that), I discovered when I found the bug, that the egress queue was set to one interface on "br-`ex`", but since sometimes we deploy with several interfaces there (for other purposes), the right one didn't get it. If that's the case -I wasn't able to follow the code properly- may be we will need to setup a list of interfaces like : external_ids:qos_egress_ifaces=<some device>,<another-device>,..." ? I'll have to take another look through the code to see if I can trace where that might be happening. From what I could tell, the only way egress qos qdiscs were set up was for tunnels. It may be that somehow an egress tunnel iface is being set up incorrectly by OVS? I'm not sure. Your suggestion could work, too. It might be more efficient than my idea as well. I'll provide a patch for you to test and you can tell me if it works. Hi Miguel. I've created a branch in my fork of OVS where I've attempted what I described earlier. I would appreciate if you could give this a test in your setup to see if it works as expected. https://github.com/putnopvut/ovs/tree/qos-thing The method I ended up going with is similar to what I initially discussed, but I decided to make it slightly more generic. On br-ex, you can set 'external-ids:ovn-egress-iface=true' on each interface that should have QoS qdiscs enabled. I have not tested this myself yet, so your feedback will be valuable. Here's a scratch build of OVN 2.9 I made with the patch applied. Hopefully one of the links below will be useful for testing. To reiterate how this is intended to work, in the bridge that br-int is connected to, you can specify on the egress interface "external-ids:ovn-egress-interface=true" and we should set up QoS on that interface. Once again, this has not been tested yet because I am not sure about the proper environment to test it in. So this may not work as expected. A yum repository for the scratch build of openvswitch-2.9.0-81.el7fdn.bz1580542.1 (task 19171354) is available at: http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory: http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/openvswitch-2.9.0-81.el7fdn.bz1580542.1-scratch.repo RPMs and build logs can be found in the following locations: http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/ http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/ http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/ http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/ The full list of available rpms is: http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/aarch64/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.aarch64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/ppc64le/openvswitch-2.9.0-81.el7fdn.bz1580542.1.ppc64le.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/s390x/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.s390x.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.src.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/python-openvswitch-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-test-2.9.0-81.el7fdn.bz1580542.1.noarch.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-debuginfo-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-central-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-common-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-vtep-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-devel-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm http://brew-task-repos.usersys.redhat.com/repos/scratch/mmichels/openvswitch/2.9.0/81.el7fdn.bz1580542.1/x86_64/openvswitch-ovn-host-2.9.0-81.el7fdn.bz1580542.1.x86_64.rpm Verified on the latest version:
[root@dell-per730-42 ovn]# uname -a
Linux dell-per730-42.rhts.eng.pek2.redhat.com 3.10.0-1101.el7.x86_64 #1 SMP Sat Oct 5 04:50:26 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per730-42 ovn]# rpm -qa | grep ovn
ovn2.11-host-2.11.1-8.el7fdp.x86_64
ovn2.11-central-2.11.1-8.el7fdp.x86_64
ovn2.11-2.11.1-8.el7fdp.x86_64
[root@dell-per730-42 ovn]# rpm -qa | grep openvswitch
kernel-kernel-networking-openvswitch-ovs_qinq-1.3-34.noarch
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
openvswitch2.11-2.11.0-26.el7fdp.x86_64
[root@dell-per730-42 ovn]#
[root@dell-per730-42 ovn]# ovn-nbctl show
switch 107877ac-001e-451d-be65-88346a2793eb (s3)
port hv0_vm00_vnet1
addresses: ["00:de:ad:00:00:01 172.16.103.11"]
port s3_r1
type: router
addresses: ["00:de:ad:ff:01:03 172.16.103.1"]
router-port: r1_s3
port hv0_vm01_vnet1
addresses: ["00:de:ad:00:01:01 172.16.103.12"]
switch eff2f042-f1dc-412f-867f-b414e0934acb (public)
port public_r1
type: router
router-port: r1_public
port ln_p1
type: localnet
addresses: ["unknown"]
switch b4b3a93e-12f4-429b-ba6e-d2d223241fb6 (s2)
port hv1_vm01_vnet1
addresses: ["00:de:ad:01:01:01 172.16.102.12"]
port s2_r1
type: router
addresses: ["00:de:ad:ff:01:02 172.16.102.1"]
router-port: r1_s2
port hv1_vm00_vnet1
addresses: ["00:de:ad:01:00:01 172.16.102.11"]
router a2c242a1-de48-4c0a-ba52-a592642d9767 (r1)
port r1_s3
mac: "00:de:ad:ff:01:03"
networks: ["172.16.103.1/24"]
port r1_public
mac: "40:44:00:00:00:03"
networks: ["172.16.104.1/24"]
port r1_s2
mac: "00:de:ad:ff:01:02"
networks: ["172.16.102.1/24"]
nat 634324ee-4c29-4df6-a9c2-d4fbc3069982
external ip: "172.16.104.200"
logical ip: "172.16.102.11"
type: "dnat_and_snat"
nat ef54f72d-14dd-4010-8e63-315fe3b7332f
external ip: "172.16.104.201"
logical ip: "172.16.103.11"
type: "dnat_and_snat"
[root@dell-per730-42 ovn]#
add qos command with the commands:
ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_max_rate=1000
ovn-nbctl set Logical_Switch_Port ln_p1 options:qos_burst=1000
ovs-vsctl set interface p5p2 external-ids:ovn-egress-iface=true
Then the traffic is low:
[root@dell-per730-42 ovn]# virsh console hv1_vm00
Connected to domain hv1_vm00
Escape character is ^]
[root@localhost ~]# netperf -H 172.16.104.201
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.104.201 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 14.00 0.01
[root@localhost ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3718 |
Description of problem: This is the U/S version of the bug: https://github.com/openvswitch/ovs-issues/issues/150 When a port has assigned qos_max_rate / qos_burst and at the same time the port is directly attached to a provider network logical switch (with a physical mapping), or it has a distributed floating IP (which won't go through a tunnel to a gateway chassis), the bandwidth limit has no effect since the queues are only configured to tunnel ports, and not also on physical interfaces of the bridge mappings. Version-Release number of selected component (if applicable): 2.9.0 How reproducible: Always