Description of problem: I am using RHOSP 16.2.3 Created openstack neutron port VM openstack port create --network vlan1 --vnic-type normal --mac-address f8:f2:1e:03:bf:f6 vlanport2 openstack port set --no-security-group --disable-port-security vlanport2 openstack server create --flavor F1 --image Testpmd --nic port-id=b6f1cc14-6264-4f7b-a1ef-f07a8805a749 --key-name test --availability-zone nova:hareshcomputesriov-0.localdomain VM2 Create neutron policy and applied on neutron port openstack network qos policy create QP1 openstack network qos rule create --type minimum-bandwidth --min-kbps 4000000 --egress QP1 openstack port set --qos-policy QP1 vlanport2 Queue configured on the port [root@hareshcomputesriov-0 heat-admin]# ovs-vsctl list queue _uuid : 6869e6fb-6356-41cd-a983-51b7a513d47a dscp : [] external_ids : {port="b6f1cc14-6264-4f7b-a1ef-f07a8805a749", queue-num="6", type=minimum_bandwidth} other_config : {min-rate="4000000000"} [root@hareshcomputesriov-0 heat-admin]# ovs-vsctl list qos _uuid : f1d91cec-e1ac-402d-9222-a8e3ed9a0b08 external_ids : {_type=minimum_bandwidth, id="7dabaec4-d6dc-49de-b995-54a830d9d184"} other_config : {} queues : {6=6869e6fb-6356-41cd-a983-51b7a513d47a} type : linux-htb However, egress bond interface has ceil rate set to 100Mbit which limits traffic of this VirtIO port to 100 mbps. [root@hareshcomputesriov-0 /]# tc class show dev mx-bond class htb 1:1 parent 1:fffe prio 0 rate 72Kbit ceil 100Mbit burst 9063b cburst 9062b class htb 1:fffe root rate 100Mbit ceil 100Mbit burst 9000b cburst 9000b class htb 1:7 parent 1:fffe prio 0 rate 100Mbit ceil 100Mbit burst 9062b cburst 9062b [root@hareshcomputesriov-0 heat-admin]# ethtool mx-bond Settings for mx-bond: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 50000Mb/s Duplex: Full Auto-negotiation: off Port: Other PHYAD: 0 Transceiver: internal Link detected: yes Ovs flows: Egress direction: ufid:de5f037f-e7e9-4533-9417-89682b2322da, recirc_id(0),dp_hash(0/0),skb_priority(0),in_port(qvob6f1cc14-62),skb_mark(0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=f8:f2:1e:03:bf:f6,dst=c2:64:5c:2b:8f:75),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:316457, bytes:3552789956, used:0.037s, flags:SP., dp:ovs, actions:push_vlan(vid=415,pcp=0),set(skb_priority(0x10007)),set(skb_mark(0x6)),mx-bond Ingress direction: ufid:8cce6272-fcaf-48cf-ba4d-7e9fe2f21f4a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(mx-bond),packet_type(ns=0/0,id=0/0),eth(src=c2:64:5c:2b:8f:75,dst=f8:f2:1e:03:bf:f6),eth_type(0x8100),vlan(vid=415,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no)), packets:2633183, bytes:158008104, used:0.000s, dp:tc, actions:pop_vlan,qvob6f1cc14-62 _uuid : 05831030-c3c0-4b1a-bcc6-ac4e14470e2f bond_active_slave : [] bond_downdelay : 0 bond_fake_iface : false bond_mode : [] bond_updelay : 0 cvlans : [] external_ids : {} fake_bridge : false interfaces : [b1e01f51-3d55-402e-aac7-24e28ba95dbc] lacp : [] mac : [] name : mx-bond other_config : {} protected : false qos : f1d91cec-e1ac-402d-9222-a8e3ed9a0b08 rstp_statistics : {} rstp_status : {} statistics : {} status : {} tag : [] trunks : [] vlan_mode : [] [root@hareshcomputesriov-0 heat-admin]# cat /proc/net/bonding/mx-bond Ethernet Channel Bonding Driver: v4.18.0-305.45.1.el8_4.x86_64 Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 Slave Interface: enp4s0f0 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 04:3f:72:d9:c0:48 Slave queue ID: 0 Slave Interface: enp4s0f1 MII Status: up Speed: 25000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 04:3f:72:d9:c0:49 Slave queue ID: 0 [root@hareshcomputesriov-0 heat-admin]# Openstack port has minimum bandwidth rate configure properly. (overcloud) [stack@undercloud-0 ~]$ openstack port show vlanport2 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | hareshcomputesriov-0.localdomain | | binding_profile | | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2022-06-02T13:20:18Z | | data_plane_status | None | | description | | | device_id | cfc567f0-f924-4c2e-bacb-9a52314f6f3b | | device_owner | compute:nova | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='6.6.6.98', subnet_id='8f75cee1-99f9-4767-8b29-b10b84c4726a' | | id | b6f1cc14-6264-4f7b-a1ef-f07a8805a749 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='84834548a1c841299a5425484a907ff9', project.name='admin', region_name='regionOne', zone= | | mac_address | f8:f2:1e:03:bf:f6 | | name | vlanport2 | | network_id | 4d026cb6-2e9b-4c6f-b566-6a8a6f30b119 | | port_security_enabled | False | | project_id | 84834548a1c841299a5425484a907ff9 | | propagate_uplink_status | None | | qos_policy_id | 1d19879c-952d-4661-a74e-bef847629650 | | resource_request | {'required': ['CUSTOM_PHYSNET_DATA', 'CUSTOM_VNIC_TYPE_NORMAL'], 'resources': {'NET_BW_EGR_KILOBIT_PER_SEC': 4000000}} | | revision_number | 6 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2022-06-02T13:51:31Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Version-Release number of selected component (if applicable): RHEL: 8.4 Kernel: 4.18.0-305.45.1.el8_4.x86_64 ovs: 2.15.4 (DB schema 8.2.0) How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: wrongly throttling egress traffic for the port Expected results: Guaranteed bandwidth should be honored. Additional info: Note: i observed same behavior with mlx representor port as well.
Hello: I've reviewed how "tc" works and the problem is not in OVS but in how Neutron sets the values for the "tc" parent class and the queues (child classes of the root one). What OVS creates is a linux-htb class tree: class htb 1:1 parent 1:fffe prio 0 rate 24Kbit ceil 34359Mbit burst 3Kb cburst 0b class htb 1:fffe root rate 34359Mbit ceil 34359Mbit burst 0b cburst 0b class htb 1:7 parent 1:fffe prio 0 rate 2Gbit ceil 10Gbit burst 3000b cburst 2500b "1:fffe" is the root class and the other ones depend on the queues (in this case, there is a queue with a child class associated "1:7"). There are two considerations we need to change in Neutron: - The parent class *must" have enough BW for the child classes. That means we need to create the QoS register with the maximum possible value for other_config:max-rate=2^32-1. - A class cannot exceed the rate value (min-bw) over the ceil one (max-bw). Because we are using those QoS/queues registers only for min-BW, we need to create those childs classes (queue registers) with, again, other_config:max-rate=2^32-1. I'll swap this BZ to Neutron. Regards.
Verified. OpenStack version: (overcloud) [stack@undercloud-0 ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 16.2.3 (Train) Network: (overcloud) [stack@undercloud-0 ~]$ openstack network show mellanox-vlan-provider +---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2022-06-15T17:43:16Z | | description | | | dns_domain | None | | id | 2271fb39-7a53-400a-8e06-e4365fee0146 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | location | cloud='', project.domain_id=, project.domain_name=, project.id='f96ce30d42ba419cad43c73e86d2fd16', project.name=, region_name='regionOne', zone= | | mtu | 9000 | | name | mellanox-vlan-provider | | port_security_enabled | False | | project_id | f96ce30d42ba419cad43c73e86d2fd16 | | provider:network_type | vlan | | provider:physical_network | mx-network | | provider:segmentation_id | 136 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 4562eeee-a102-4687-871e-2767a355945d | | tags | | | updated_at | 2022-06-15T17:43:17Z | +---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ Create port (type normal) in network mellanox-vlan-provider: (venv) (overcloud) [stack@undercloud-0 ~]$ openstack port create --network mellanox-vlan-provider --vnic-type normal --mac-address fa:ba:da:fa:ba:da testport0 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2022-06-15T18:43:26Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | id | 860a57d0-21e1-48d4-9bf3-e2e2647880e0 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='abbc5d1189234e19a039e6787ce36db6', project.name='admin', region_name='regionOne', zone= | | mac_address | fa:ba:da:fa:ba:da | | name | testport0 | | network_id | 2271fb39-7a53-400a-8e06-e4365fee0146 | | port_security_enabled | False | | project_id | abbc5d1189234e19a039e6787ce36db6 | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 1 | | security_group_ids | | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2022-06-15T18:43:26Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Create VM with NIC testport0: (overcloud) [stack@undercloud-0 ~]$ openstack server create --image rhel-guest-image-7.6-210.x86_64.qcow2 --flavor nfv_qe_base_flavor --nic port-id=860a57d0-21e1-48d4-9bf3-e2e2647880e0 testvm0 --wait +-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | computehwoffload-1.redhat.local | | OS-EXT-SRV-ATTR:hostname | testvm0 | | OS-EXT-SRV-ATTR:hypervisor_hostname | computehwoffload-1.redhat.local | | OS-EXT-SRV-ATTR:instance_name | instance-00000096 | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-956ce5nb | | OS-EXT-SRV-ATTR:root_device_name | /dev/vda | | OS-EXT-SRV-ATTR:user_data | None | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2022-06-15T18:48:55.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | adminPass | yL6SCqmUB5Ro | | config_drive | | | created | 2022-06-15T18:48:48Z | | description | None | | flavor | disk='20', ephemeral='0', extra_specs.hw:cpu_policy='dedicated', extra_specs.hw:emulator_threads_policy='share', extra_specs.hw:mem_page_size='large', original_name='nfv_qe_base_flavor', ram='8192', swap='0', vcpus='6' | | hostId | 31b3a1f438676250e8e90ea31b0c2471d34ab53bf28595268de41858 | | host_status | UP | | id | 6be204fb-5e10-4b78-8c0e-5e5307886b14 | | image | rhel-guest-image-7.6-210.x86_64.qcow2 (d0712183-6959-4c11-9f95-16cf02409597) | | key_name | None | | locked | False | | locked_reason | None | | name | testvm0 | | progress | 0 | | project_id | abbc5d1189234e19a039e6787ce36db6 | | properties | | | security_groups | name='default' | | server_groups | [] | | status | ACTIVE | | tags | [] | | trusted_image_certificates | None | | updated | 2022-06-15T18:48:55Z | | user_id | b49fbd3d0e0e40de84a93bdc41f7a222 | | volumes_attached | | +-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ On compute, there is no OVS minimum configuration as there is no network policy configured yet: [root@computehwoffload-1 ~]# ovs-vsctl list qos [root@computehwoffload-1 ~]# ovs-vsctl list queue [root@computehwoffload-1 ~]# tc class show dev mx-bond Create network qos policy with egress rule of type minimum-bandwidth: (venv) (overcloud) [stack@undercloud-0 ~]$ openstack network qos policy create QP0 +-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | description | | | id | ff88f2c9-6551-4996-b252-cd720e16f363 | | is_default | False | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='abbc5d1189234e19a039e6787ce36db6', project.name='admin', region_name='regionOne', zone= | | name | QP0 | | project_id | abbc5d1189234e19a039e6787ce36db6 | | rules | [] | | shared | False | | tags | [] | +-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (venv) (overcloud) [stack@undercloud-0 ~]$ openstack network qos rule create --type minimum-bandwidth --min-kbps 4000000 --egress QP0 +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | direction | egress | | id | 06f140a0-c23a-4b63-bc18-84b75eef9684 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='abbc5d1189234e19a039e6787ce36db6', project.name='admin', region_name='regionOne', zone= | | min_kbps | 4000000 | | name | None | | project_id | | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (venv) (overcloud) [stack@undercloud-0 ~]$ openstack port set --qos-policy QP0 testport0 (venv) (overcloud) [stack@undercloud-0 ~]$ openstack port show testport0 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-1.redhat.local | | binding_profile | | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2022-06-15T18:43:26Z | | data_plane_status | None | | description | | | device_id | 6be204fb-5e10-4b78-8c0e-5e5307886b14 | | device_owner | compute:nova | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | id | 860a57d0-21e1-48d4-9bf3-e2e2647880e0 | | location | cloud='', project.domain_id=, project.domain_name='Default', project.id='abbc5d1189234e19a039e6787ce36db6', project.name='admin', region_name='regionOne', zone= | | mac_address | fa:ba:da:fa:ba:da | | name | testport0 | | network_id | 2271fb39-7a53-400a-8e06-e4365fee0146 | | port_security_enabled | False | | project_id | abbc5d1189234e19a039e6787ce36db6 | | propagate_uplink_status | None | | qos_policy_id | ff88f2c9-6551-4996-b252-cd720e16f363 | | resource_request | {'required': ['CUSTOM_PHYSNET_MX_NETWORK', 'CUSTOM_VNIC_TYPE_NORMAL'], 'resources': {'NET_BW_EGR_KILOBIT_PER_SEC': 4000000}} | | revision_number | 10 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2022-06-15T18:53:17Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+ OVS queue configuration properly configured after setting the network policy: [root@computehwoffload-1 ~]# ovs-vsctl list queue _uuid : 7bb53dad-ab18-421d-b822-d0b92f8c4a28 dscp : [] external_ids : {port="860a57d0-21e1-48d4-9bf3-e2e2647880e0", queue-num="16", type=minimum_bandwidth} other_config : {max-rate="34359738367", min-rate="4000000000"} [root@computehwoffload-1 ~]# ovs-vsctl list qos _uuid : 11491728-fd9a-4767-afd7-3ad476b55d40 external_ids : {_type=minimum_bandwidth, id="f7386ecb-3628-4a9f-bc39-ba06b71d183d"} other_config : {max-rate="34359738367"} queues : {16=7bb53dad-ab18-421d-b822-d0b92f8c4a28} type : linux-htb Also, the bond interface has properly configured the ceil rate to 4Gbit: [root@computehwoffload-1 ~]# tc class show dev mx-bond class htb 1:11 parent 1:fffe prio 0 rate 4Gbit ceil 34359Mbit burst 9000b cburst 8589b class htb 1:1 parent 1:fffe prio 0 rate 72Kbit ceil 34359Mbit burst 9063b cburst 8589b class htb 1:fffe root rate 34359Mbit ceil 34359Mbit burst 8589b cburst 8589b [root@computehwoffload-1 ~]# ethtool mx-bond Settings for mx-bond: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 40000Mb/s Duplex: Full Auto-negotiation: off Port: Other PHYAD: 0 Transceiver: internal Link detected: yes
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:4793