Bug 2069742
| Summary: | QoS BW limit rule not applied after updating its value | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Eduardo Olivares <eolivare> |
| Component: | ovn-2021 | Assignee: | lorenzo bianconi <lorenzo.bianconi> |
| Status: | CLOSED ERRATA | QA Contact: | ying xu <yinxu> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | FDP 22.C | CC: | ctrautma, jiji, mmichels |
| Target Milestone: | --- | Keywords: | AutomationBlocker, Regression |
| Target Release: | FDP 22.C | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ovn-2021-21.12.0-42.el8fdp | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-04-25 14:26:54 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi Eduardo, Can you attach the OVN northbound and southbound DBs from this scenario? This will help us to reproduce the failure outside an OSP environment and drill down into which commit caused the regression. Thanks. I have tried about it ,but I can't reproduce it.
# ovn-nbctl list logical-switch s2
_uuid : db9bb9c4-d437-4b7c-94a6-3a2fe1e0df44
acls : []
copp : []
dns_records : []
external_ids : {}
forwarding_groups : []
load_balancer : []
load_balancer_group : []
name : s2
other_config : {ipv6_prefix="2001:db8:102::0", subnet="172.16.102.0/24"}
ports : [1083b8c8-0340-4146-9e3e-7a9df2cd832e, 3a28d8ca-f19b-4bfc-b3aa-f2a45be6a734, 6af8d979-6a54-4965-8fe2-613d26ae0433, 6f53c127-3c5c-42ee-b3e0-1ba67b861a0d, 9c44b03f-f523-4346-95eb-14ff5af1173a]
qos_rules : [c12edd48-6ac7-4082-bf49-0d3837f3a60e]
# ovn-nbctl list qos
_uuid : c12edd48-6ac7-4082-bf49-0d3837f3a60e
action : {dscp=48}
bandwidth : {burst=400000, rate=500000}
direction : to-lport
external_ids : {}
match : "outport==\"hv0_vm00_vnet1\""
priority : 100
(In reply to lorenzo bianconi from comment #4) > upstream fix: > https://patchwork.ozlabs.org/project/ovn/patch/ > 3ab569f05ec47bff464aed29d6c7610384603224.1648847020.git.lorenzo. > bianconi/ I have tested this patch and the osp qos-bwlimit tests originally affected now pass. The meters are successfullu created, updated and removed during the bwlimit tests. reproduced on version: # rpm -qa|grep ovn ovn-2021-central-21.12.0-37.el8fdp.x86_64 ovn-2021-host-21.12.0-37.el8fdp.x86_64 ovn-2021-21.12.0-37.el8fdp.x86_64 This was reproduced by changing the rate many times: [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=1000000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] Red Hat Enterprise Linux Server 7.8 Beta (Maipo) Kernel 3.10.0-1118.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Mon Apr 11 00:05:40 on ttyS0 [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.03 1135.81 -------------------------it's correct [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=500000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.04 588.08 ---------------------------it's correct when change only once [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=300000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=600000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=500000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.03 1135.56 -------------------------------it's wrong after change 4 times. verified on version: # rpm -qa|grep ovn ovn-2021-21.12.0-42.el8fdp.x86_64 ovn-2021-central-21.12.0-42.el8fdp.x86_64 ovn-2021-host-21.12.0-42.el8fdp.x86_64 change many times, every time works fine! # ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=1000000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] Red Hat Enterprise Linux Server 7.8 Beta (Maipo) Kernel 3.10.0-1118.el7.x86_64 on an x86_64 localhost login: root Password: Last login: Mon Apr 11 01:28:39 on ttyS0 [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.02 1132.36 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=500000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.04 580.71 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=300000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=600000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.01 688.33 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=500000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=700000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=1000000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=500000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.01 584.25 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=700000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=300000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=400000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=800000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.02 911.03 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=400000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.01 475.14 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=600000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=1000000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 1133.23 [root@localhost ~]# [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=700000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=300000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=500000,burst=400000 [root@dell-per740-53 basic]# ovn-nbctl --wait=hv set QoS $qos_id bandwidth=rate=700000,burst=400000 [root@dell-per740-53 basic]# virsh console hv1_vm00 Connected to domain hv1_vm00 Escape character is ^] [root@localhost ~]# netperf -4 -H 172.16.102.21 -t TCP_STREAM -T1 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.102.21 () port 0 AF_INET : cpu bind Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.01 796.62 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ovn-2021 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1523 |
Description of problem: We are testing ovn-2021-21.12.0-37 on Openstack and we have found this regression with some qos tempest tests. The following qos bwlimit tests fail: test_bw_limit_south_north test_bw_limit_east_west test_bw_limit_tenant_network Link to logs from one of these tests: [1]. Search for measured_bw. The first configured bw limit is correctly applied: 2022-03-28 19:48:47,968 392485 DEBUG [neutron_plugin.tests.scenario.test_qos] egress udp / server_ip = 192.168.1.60 / measured_bw = 999243.0591640397 2022-03-28 19:48:47,968 392485 DEBUG [neutron_plugin.tests.scenario.test_qos] egress bw_limit = 1000000 Then, the qos BW limit rule is updated, but the new value is ignored: 2022-03-28 19:49:00,469 392485 DEBUG [neutron_plugin.tests.scenario.test_qos] egress tcp / server_ip = 192.168.1.60 / measured_bw = 995680.9759319122 2022-03-28 19:49:00,469 392485 DEBUG [neutron_plugin.tests.scenario.test_qos] egress bw_limit = 500000 These tests pass on ovn-2021-21.12.0-11, so this seems to be a regression. I have checked the NBDB on ovn-2021-21.12.0-37 and it seems the DB is provisioned with the proper values: [root@controller-0 ~]# ovn-nbctl get logical_switch tempest-test-network--180258014 qos [4adf5959-e232-4635-a223-0ba580dad13b, 9ae674af-66b4-49ce-8ed1-b8570a7b2226] [root@controller-0 ~]# ovn-nbctl list logical_switch tempest-test-network--180258014 _uuid : ecf7a078-aa77-4b28-a7e8-c739da418cce acls : [] copp : [] dns_records : [f6f86d9b-62c2-4ce2-bc41-0a937b0328d8] external_ids : {"neutron:mtu"="8942", "neutron:network_name"=tempest-test-network--180258014, "neutron:revision_number"="3 "} forwarding_groups : [] load_balancer : [] load_balancer_group : [] name : neutron-33415ce6-5003-44f9-9f2d-cc8089d12cdb other_config : {mcast_flood_unregistered="false", mcast_snoop="true", vlan-passthru="false"} ports : [07e42ba6-1004-4989-b88b-2322d27ba05e, 41285ac1-bfda-4f60-8da7-3febc3872215, 48d4c5a8-71d1-4da2-8178-5261c5 26b8b4, 7f37df1f-945d-4e98-b7af-4faa436e0c1c] qos_rules : [4adf5959-e232-4635-a223-0ba580dad13b, 9ae674af-66b4-49ce-8ed1-b8570a7b2226] [root@controller-0 ~]# ovn-nbctl list qos 4adf5959-e232-4635-a223-0ba580dad13b _uuid : 4adf5959-e232-4635-a223-0ba580dad13b action : {} bandwidth : {burst=400, rate=500} <-- CORRECT VALUES SET 500000 direction : from-lport external_ids : {} match : "inport == \"0748f390-69aa-46b7-8435-cba00bcd4c93\"" priority : 2002 [root@controller-0 ~]# ovn-nbctl list qos 9ae674af-66b4-49ce-8ed1-b8570a7b2226 _uuid : 9ae674af-66b4-49ce-8ed1-b8570a7b2226 action : {} bandwidth : {burst=400, rate=500} <-- CORRECT VALUES SET 500000 direction : from-lport external_ids : {} match : "inport == \"3592e8b8-2418-4657-a323-d77015890f76\"" priority : 2002 From neutron API side, everything seems fine too: (overcloud) [stack@undercloud-0]$ openstack network qos policy show d840ab26-cfb8-43a7-b8e0-4fde610e0dd9 -c rules -f value [{'max_kbps': 500, 'max_burst_kbps': 400, 'direction': 'egress', 'id': '949e6f4b-00c8-4833-9c01-a423ed76b206', 'qos_policy_id': 'd840ab26-cfb8-43a7-b8e0-4fde610e0dd9', 'type': 'bandwidth_limit'}] Then, I'm running manually iperf on the VMs and the BW exceeds the configured 500kbps by far. # On server VM side $ iperf3 -s -p 4321 # On client VM $ iperf3 -c 192.168.0.140 -p 4321 -t 10 -b 20000000 [1] https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-networking-ovn-16.2_director-rhel-virthost-3cont_2comp_3net-ipv4-geneve-composable-vlan-provider-network/83/testReport/neutron_plugin.tests.scenario.test_qos/QosTestCommon/test_bw_limit_east_west_id_7ce63fb6_b2c4_4b14_8aee_9e8304ad7d3d_/ Version-Release number of selected component (if applicable): ovn-2021-21.12.0-37 + RHOS-16.2-RHEL-8-20220311.n.1 How reproducible: 100% Steps to Reproduce: 1. create a qos policy, a qos rule with bwlimit, assign it to a network and create two VMs on that network (cirros is not valid, use RHEL) 2. run iperf3 on those VMs as described above 3.