Bug 2092946
| Summary: | [RHOSP16.2] ICMP/TCP/UDP Flows Are Not Offloaded In OVS Based Deployment | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Vadim Khitrin <vkhitrin> |
| Component: | openstack-neutron | Assignee: | Rodolfo Alonso <ralonsoh> |
| Status: | CLOSED ERRATA | QA Contact: | Eran Kuris <ekuris> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 16.2 (Train) | CC: | apevec, bcafarel, broose, chrisw, gregraka, hakhande, kgilliga, marjones, mburns, mleitner, mnietoji, mtomaska, oblaut, pgrist, ralonsoh, rdiazcam, scohen, slinaber, spower, stchen, supadhya, vchundur |
| Target Milestone: | z3 | Keywords: | Regression, Triaged |
| Target Release: | 16.2 (Train on RHEL 8.4) | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-neutron-15.3.5-2.20220413175559.3810153.el8ost | Doc Type: | Release Note |
| Doc Text: |
Open vSwitch (OVS) does not support offloading OpenFlow rules that have the `skb_priority`, `skb_mark`, or output queue fields set. Those fields are needed to provide quality-of-service (QoS) support for virtio ports.
+
If you set a minimum bandwidth rule for a virtio port, the Neutron Open vSwitch agent marks the traffic of this port with a Packet Mark Field. As a result, this traffic cannot be offloaded, and it affects the traffic in other ports. If you set a bandwidth limit rule, all traffic is marked with the default 0 queue, which means no traffic can be offloaded.
+
As a workaround, if your environment includes OVS hardware offload ports, disable the packet marking in the nodes that require hardware offloading. After you disable the packet marking, it will not be possible to set rate limiting rules for virtio ports. However, differentiated services code point (DSCP) marking rules will still be available.
+
In the configuration file, set the `disable_packet_marking` flag to `true`. After you edit the configuration file, you must restart the `neutron_ovs_agent` container.
For example:
+
----
$ cat `/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini`
[ovs]
disable_packet_marking=True
----
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-22 16:07:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1846101, 2018001 | ||
is it a regression? Hello all: This is not a regression for OVS without HW offload. The patch that introduced this action is [1], that is needed to make ingress BW limit and egress minimum BW rules work together. We can't remove this marking action. For OVS with HW offload it could be consider that. If the HW offloaded OVS does not support any kind of QoS (if I'm not wrong, in OVS with HW offloaded, the metering action does not work yet), what I suggest is to filter out (in openvswitch) any queue set on the flows and process it. Regards. [1]https://review.opendev.org/q/I1e31565475f38c6ad817268699b165759ac05410 (In reply to Haresh Khandelwal from comment #5) > (In reply to Rodolfo Alonso from comment #4) > > Hello all: > > > > This is not a regression for OVS without HW offload. The patch that > > introduced this action is [1], that is needed to make ingress BW limit and > > egress minimum BW rules work together. We can't remove this marking action. > > > > For OVS with HW offload it could be consider that. If the HW offloaded OVS > > does not support any kind of QoS (if I'm not wrong, in OVS with HW > > offloaded, the metering action does not work yet), what I suggest is to > > filter out (in openvswitch) any queue set on the flows and process it. > > > > Regards. > > > > > > [1]https://review.opendev.org/q/I1e31565475f38c6ad817268699b165759ac05410 > > Hi Rodolfo, > > we should consider this as a regression for ovs-hw-offload. We have to clear > out skb priority from action. This is non offload-able action. > > 2022-06-05T17:24:50.224Z|00001|netdev_offload_tc(handler2)|DBG|offloading > attribute skb_priority isn't supported > 2022-06-05T17:24:50.224Z|00005|netdev_offload_tc(handler1)|DBG|vlan_id[0]: > 415 > 2022-06-05T17:24:50.224Z|00006|netdev_offload_tc(handler1)|DBG|vlan_prio[0]: > 0 > 2022-06-05T17:24:50.224Z|00007|netdev_offload_tc(handler1)|DBG|offloading > attribute skb_priority isn't supported > > Thanks In case it's a regression I would recommend you to raise it as a blocker. Eran, this is a regression for hardware offload since it was working in z2. Raising blocker flag. I can confirm that this is working when adding 'disable_packet_marking' under 'ovs' in '/var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini'. Is there a corresponding puppet/TripleO patch to populate this entry during deployment? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.3 (Train)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:4793 |
Description of problem: Flows that are supposed to be offloaded using tc-flower are not offloaded. 2 compute nodes with Mellanox Connect-X5 NICs are used. Spawning 2 VMs: +--------------------------------------+------------------------------------------+--------+-------------------------------------------------------------------------------------------+---------------------------------------+--------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------------------+--------+-------------------------------------------------------------------------------------------+---------------------------------------+--------------------+ | 39f286e7-90b8-40f7-b629-8ee6082fb666 | tempest-TestNfvOffload-server-401965697 | ACTIVE | mellanox-vlan-provider=30.30.220.175; mellanox-vxlan-provider=20.20.220.119, 10.35.185.23 | rhel-guest-image-7-6-210-x86-64-qcow2 | nfv_qe_base_flavor | | 4d28f3c9-d175-426f-8676-1998989db656 | tempest-TestNfvOffload-server-1163462706 | ACTIVE | mellanox-vlan-provider=30.30.220.106; mellanox-vxlan-provider=20.20.220.191, 10.35.185.26 | rhel-guest-image-7-6-210-x86-64-qcow2 | nfv_qe_base_flavor | +--------------------------------------+------------------------------------------+--------+-------------------------------------------------------------------------------------------+---------------------------------------+--------------------+ Ports are configured in switched mode: openstack port show 167e7e26-5002-4425-a2af-c67cf0cc854e +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | computehwoffload-0.redhat.local | | binding_profile | capabilities='['switchdev']', pci_slot='0000:07:01.1', pci_vendor_info='15b3:1018', physical_network= | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2022-06-02T14:40:12Z | | data_plane_status | None | | description | | | device_id | 39f286e7-90b8-40f7-b629-8ee6082fb666 | | device_owner | compute:nova | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='20.20.220.119', subnet_id='474f8fa2-a8ff-4ea8-a60b-4d0f94c50bfe' | | id | 167e7e26-5002-4425-a2af-c67cf0cc854e | | location | cloud='', project.domain_id=, project.domain_name=, project.id='c3ecb6bc81df459d86728598a118dac2', project.name=, region_name='regionOne', zone= | | mac_address | fa:16:3e:1e:fa:d6 | | name | tempest-port-smoke-1387278248 | | network_id | 394da388-080b-41ef-b036-66bb102c1eb5 | | port_security_enabled | False | | project_id | c3ecb6bc81df459d86728598a118dac2 | | propagate_uplink_status | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2022-06-02T14:40:59Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ Pinging (ICMP) and sending TCP/UDP traffic we notice the flows are not offloaded: ping 20.20.220.119 PING 20.20.220.119 (20.20.220.119) 56(84) bytes of data. 64 bytes from 20.20.220.119: icmp_seq=1 ttl=64 time=0.107 ms 64 bytes from 20.20.220.119: icmp_seq=2 ttl=64 time=0.127 ms 64 bytes from 20.20.220.119: icmp_seq=3 ttl=64 time=0.129 ........ [root@computehwoffload-0 ~]# ovs-appctl dpctl/dump-flows recirc_id(0),in_port(2),eth(src=f4:52:14:25:28:4a,dst=01:80:c2:00:00:0e),eth_type(0x88cc), packets:0, bytes:0, used:3.770s, actions:drop tunnel(tun_id=0x9569,src=10.10.111.152,dst=10.10.111.195,tp_dst=4789,flags(+key)),recirc_id(0),in_port(9),eth(),eth_type(0x0800),ipv4(frag=no), packets:3855, bytes:208170, used:0.690s, actions:drop recirc_id(0),skb_priority(0),in_port(10),eth(src=fa:16:3e:1e:fa:d6,dst=fa:16:3e:62:57:6f),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:325, bytes:29143, used:3.060s, flags:P., actions:set(tunnel(tun_id=0x920f,src=10.10.111.195,dst=10.10.111.152,ttl=64,tp_dst=4789,flags(df|key))),set(skb_priority(0x10001)),9 recirc_id(0),in_port(2),eth(src=f4:52:14:25:28:4a,dst=01:80:c2:00:00:00),eth_type(0/0xffff), packets:3771, bytes:226260, used:0.711s, actions:drop recirc_id(0),skb_priority(0),tunnel(tun_id=0x920f,src=10.10.111.152,dst=10.10.111.195,flags(-df-csum+key)),in_port(9),eth(src=fa:16:3e:62:57:6f,dst=fa:16:3e:1e:fa:d6),eth_type(0x0806), packets:0, bytes:0, used:never, actions:set(skb_priority(0x10001)),10 recirc_id(0),skb_priority(0),tunnel(tun_id=0x920f,src=10.10.111.152,dst=10.10.111.195,flags(-df-csum+key)),in_port(9),eth(src=fa:16:3e:62:57:6f,dst=fa:16:3e:1e:fa:d6),eth_type(0x0800),ipv4(frag=no), packets:352, bytes:28365, used:3.060s, flags:P., actions:set(skb_priority(0x10001)),10 recirc_id(0),skb_priority(0),in_port(10),eth(src=fa:16:3e:1e:fa:d6,dst=fa:16:3e:62:57:6f),eth_type(0x0806), packets:0, bytes:0, used:never, actions:set(tunnel(tun_id=0x920f,src=10.10.111.195,dst=10.10.111.152,ttl=64,tp_dst=4789,flags(df|key))),set(skb_priority(0x10001)),9 [root@computehwoffload-1 ~]# ovs-appctl dpctl/dump-flows tunnel(tun_id=0x9569,src=10.10.111.152,dst=10.10.111.166,tp_dst=4789,flags(+key)),recirc_id(0),in_port(5),eth(),eth_type(0x0800),ipv4(frag=no), packets:3593, bytes:194022, used:0.020s, actions:drop recirc_id(0),skb_priority(0),in_port(2),eth(src=fa:16:3e:fa:18:bf,dst=fa:16:3e:62:57:6f),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:1180, bytes:188689, used:5.047s, flags:SP., actions:set(tunnel(tun_id=0x920f,src=10.10.111.166,dst=10.10.111.152,ttl=64,tp_dst=4789,flags(df|key))),set(skb_priority(0x10001)),5 recirc_id(0),skb_priority(0),tunnel(tun_id=0x920f,src=10.10.111.152,dst=10.10.111.166,flags(-df-csum+key)),in_port(5),eth(src=fa:16:3e:62:57:6f,dst=fa:16:3e:fa:18:bf),eth_type(0x0800),ipv4(frag=no), packets:1218, bytes:86269, used:5.047s, flags:SP., actions:set(skb_priority(0x10001)),2 Version-Release number of selected component (if applicable): Compose: RHOS-16.2-RHEL-8-20220525.n.2 network-scripts-openvswitch2.15-2.15.0-87.el8fdp.x86_64 rhosp-network-scripts-openvswitch-2.15-4.el8ost.1.noarch openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch openvswitch2.15-2.15.0-87.el8fdp.x86_64 rhosp-openvswitch-2.15-4.el8ost.1.noarch How reproducible: Always Steps to Reproduce: 1. Deploy hardware offload capable deployment 2. Spawn instances 3. Send traffic between instances Actual results: Flows are not offloaded. Expected results: Flows are expected to be offloaded. Additional info: