Bug 2109597
| Summary: | HW offloading issue - trunk with VXLAN parent port offloaded, but VXLAN subport not offloaded | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Mircea Vutcovici <mircea.vutcovici> |
| Component: | openvswitch | Assignee: | Haresh Khandelwal <hakhande> |
| Status: | CLOSED NOTABUG | QA Contact: | Miguel Angel Nieto <mnietoji> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 16.2 (Train) | CC: | apevec, astillma, broose, chrisw, ekuris, fboboc, hakhande, jamsmith, jschluet, mgeary, mleitner, prgutier, ralonsoh, rosingh, scohen, vkhitrin |
| Target Milestone: | ga | Keywords: | TestOnly, Triaged |
| Target Release: | 17.1 | Flags: | hakhande:
needinfo?
(mgeary) jamsmith: needinfo? (mgeary) jamsmith: needinfo? (mgeary) jamsmith: needinfo? (hakhande) |
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | kernel-5.14.0-284.11.1.el9_2 | Doc Type: | Known Issue |
| Doc Text: |
There is a hardware (HW) limitation with CX-5. Every network traffic flow has a direction in HW, either transmit (TX) or receive (RX). If the source port of the flow is a virtual function (VF), then it is also TX flow in HW. CX-5 cannot pop VLAN on TX path, which prevents offloading the flow with pop_vlan to the HW.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-08-15 18:20:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1986511 | ||
| Bug Blocks: | 2172625 | ||
Hi, I was able to reproduce the issue with Connect X5
And I have tested that with Connect X6 the issue is NOT reproduced
Used puddle RHOS-17.1-RHEL-9-20230628.n.2
Testcase run sucessfully when configured vxlan in parent and child network
python -m testtools.run nfv_tempest_plugin.tests.scenario.test_nfv_offload.TestNfvOffload.test_offload_tcp_trunk_vlan
Ran 1 test in 2411.853s
OK
NETWORKS
(overcloud) [stack@undercloud-0 ~]$ openstack network show 354e8807-87f6-40d0-99c4-e3cc1e3f3648
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2023-07-03T09:45:46Z |
| description | |
| dns_domain | None |
| id | 354e8807-87f6-40d0-99c4-e3cc1e3f3648 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| mtu | 8950 |
| name | mellanox-vxlan-provider |
| port_security_enabled | False |
| project_id | 994c523a6b724c3298fec775380f4c6b |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 51825 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | d959a665-2e69-43ca-a613-bea1b6d6a5d4 |
| tags | |
| updated_at | 2023-07-03T09:45:47Z |
+---------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack network show 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2023-07-03T09:45:47Z |
| description | |
| dns_domain | None |
| id | 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| mtu | 8492 |
| name | trunk_network |
| port_security_enabled | False |
| project_id | 994c523a6b724c3298fec775380f4c6b |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 1986 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | 7a957ce6-4dc0-4295-a1a5-5ec968b21fbf |
| tags | |
| updated_at | 2023-07-03T09:45:47Z |
+---------------------------+--------------------------------------+
SERVERS
(overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects
+--------------------------------------+------------------------------------------+--------+------------------------------------------------------+----------------------------------------------+--------------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------------------------------+--------+------------------------------------------------------+----------------------------------------------+--------------------+
| c459d5e1-af83-4eae-8e29-f8f23155eb52 | tempest-TestNfvOffload-server-1664025114 | ACTIVE | mellanox-vxlan-provider=10.46.228.106, 20.20.220.198 | rhel-guest-image-nfv-2-8.7-1660.x86_64.qcow2 | nfv_qe_ag_flavor_1 |
| 0e611841-dc65-47a1-85b1-ee874f586079 | tempest-TestNfvOffload-server-630963766 | ACTIVE | mellanox-vxlan-provider=10.46.228.100, 20.20.220.152 | rhel-guest-image-nfv-2-8.7-1660.x86_64.qcow2 | nfv_qe_ag_flavor_0 |
+--------------------------------------+------------------------------------------+--------+------------------------------------------------------+----------------------------------------------+--------------------
+
PORTS
(overcloud) [stack@undercloud-0 ~]$ openstack port show tempest-port-smoke-1748868985
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | compute-1.localdomain |
| binding_profile | capabilities='['switchdev']', pci_slot='0000:17:01.3', pci_vendor_info='15b3:101e', physical_network= |
| binding_vif_details | bridge_name='tbr-2e0fb415-f', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | direct |
| created_at | 2023-07-03T09:45:48Z |
| data_plane_status | None |
| description | |
| device_id | c459d5e1-af83-4eae-8e29-f8f23155eb52 |
| device_owner | compute:nova |
| device_profile | None |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='20.20.220.198', subnet_id='d959a665-2e69-43ca-a613-bea1b6d6a5d4' |
| id | 06136fd2-1b58-4d32-baac-f0a4fd88f7dc |
| ip_allocation | None |
| mac_address | fa:16:3e:a9:d9:59 |
| name | tempest-port-smoke-1748868985 |
| network_id | 354e8807-87f6-40d0-99c4-e3cc1e3f3648 |
| numa_affinity_policy | None |
| port_security_enabled | False |
| project_id | 994c523a6b724c3298fec775380f4c6b |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 4 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | {'trunk_id': '2e0fb415-fd29-4388-8bd8-208c8a8016ad', 'sub_ports': [{'segmentation_id': 37, 'segmentation_type': 'vlan', 'port_id': '39fb968b-09bc-4f65-8b02-0121ec89b1fd'}]} |
| updated_at | 2023-07-03T09:46:11Z |
+-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack port show d8a42b85-fa38-4de6-8de9-672fd3f4b913 --fit-width
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | compute-0.localdomain |
| binding_profile | capabilities='['switchdev']', pci_slot='0000:17:01.2', pci_vendor_info='15b3:101e', physical_network= |
| binding_vif_details | bridge_name='tbr-7685e174-f', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | direct |
| created_at | 2023-07-03T09:45:47Z |
| data_plane_status | None |
| description | |
| device_id | 0e611841-dc65-47a1-85b1-ee874f586079 |
| device_owner | compute:nova |
| device_profile | None |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='20.20.220.152', subnet_id='d959a665-2e69-43ca-a613-bea1b6d6a5d4' |
| id | d8a42b85-fa38-4de6-8de9-672fd3f4b913 |
| ip_allocation | None |
| mac_address | fa:16:3e:aa:91:a9 |
| name | tempest-port-smoke-1511182731 |
| network_id | 354e8807-87f6-40d0-99c4-e3cc1e3f3648 |
| numa_affinity_policy | None |
| port_security_enabled | False |
| project_id | 994c523a6b724c3298fec775380f4c6b |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 4 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | {'trunk_id': '7685e174-f1b1-4305-b372-6acfb2688460', 'sub_ports': [{'segmentation_id': 37, 'segmentation_type': 'vlan', 'port_id': 'cd69a024-6865-4859-b2e6-91b423649474', |
| | 'mac_address': 'fa:16:3e:34:ed:d2'}]} |
| updated_at | 2023-07-03T09:46:00Z |
+-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack port show 39fb968b-09bc-4f65-8b02-0121ec89b1fd
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | compute-1.localdomain |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2023-07-03T09:45:49Z |
| data_plane_status | None |
| description | |
| device_id | |
| device_owner | trunk:subport |
| device_profile | None |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='50.50.220.196', subnet_id='7a957ce6-4dc0-4295-a1a5-5ec968b21fbf' |
| id | 39fb968b-09bc-4f65-8b02-0121ec89b1fd |
| ip_allocation | None |
| mac_address | fa:16:3e:47:49:52 |
| name | tempest-port-smoke-15951452 |
| network_id | 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 |
| numa_affinity_policy | None |
| port_security_enabled | False |
| project_id | 994c523a6b724c3298fec775380f4c6b |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 4 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2023-07-03T09:46:11Z |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack port show cd69a024-6865-4859-b2e6-91b423649474
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | |
| binding_host_id | compute-0.localdomain |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2023-07-03T09:45:48Z |
| data_plane_status | None |
| description | |
| device_id | |
| device_owner | trunk:subport |
| device_profile | None |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='50.50.220.175', subnet_id='7a957ce6-4dc0-4295-a1a5-5ec968b21fbf' |
| id | cd69a024-6865-4859-b2e6-91b423649474 |
| ip_allocation | None |
| mac_address | fa:16:3e:34:ed:d2 |
| name | tempest-port-smoke-1567555246 |
| network_id | 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 |
| numa_affinity_policy | None |
| port_security_enabled | False |
| project_id | 994c523a6b724c3298fec775380f4c6b |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 4 |
| security_group_ids | |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2023-07-03T09:46:00Z |
+-------------------------+--------------------------------------------------------------------------------------------------------------+
TRUNK
(overcloud) [stack@undercloud-0 ~]$ openstack network trunk show 2e0fb415-fd29-4388-8bd8-208c8a8016ad
+-----------------+------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| created_at | 2023-07-03T09:45:50Z |
| description | |
| id | 2e0fb415-fd29-4388-8bd8-208c8a8016ad |
| name | |
| port_id | 06136fd2-1b58-4d32-baac-f0a4fd88f7dc |
| project_id | 6ccccae1eb15400c9c6d2b66fb344a63 |
| revision_number | 3 |
| status | ACTIVE |
| sub_ports | port_id='39fb968b-09bc-4f65-8b02-0121ec89b1fd', segmentation_id='37', segmentation_type='vlan' |
| tags | [] |
| tenant_id | 6ccccae1eb15400c9c6d2b66fb344a63 |
| updated_at | 2023-07-03T09:46:11Z |
+-----------------+------------------------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack network trunk show 7685e174-f1b1-4305-b372-6acfb2688460
+-----------------+------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| created_at | 2023-07-03T09:45:50Z |
| description | |
| id | 7685e174-f1b1-4305-b372-6acfb2688460 |
| name | |
| port_id | d8a42b85-fa38-4de6-8de9-672fd3f4b913 |
| project_id | 6ccccae1eb15400c9c6d2b66fb344a63 |
| revision_number | 3 |
| status | ACTIVE |
| sub_ports | port_id='cd69a024-6865-4859-b2e6-91b423649474', segmentation_id='37', segmentation_type='vlan' |
| tags | [] |
| tenant_id | 6ccccae1eb15400c9c6d2b66fb344a63 |
| updated_at | 2023-07-03T09:46:00Z |
+-----------------+------------------------------------------------------------------------------------------------+
VMS
[cloud-user@tempest-testnfvoffload-server-1664025114 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc mq state UP group default qlen 1000
link/ether fa:16:3e:a9:d9:59 brd ff:ff:ff:ff:ff:ff
altname enp4s0
inet 20.20.220.198/24 brd 20.20.220.255 scope global dynamic noprefixroute eth0
valid_lft 85794sec preferred_lft 85794sec
inet6 fe80::f816:3eff:fea9:d959/64 scope link
valid_lft forever preferred_lft forever
3: eth0.37@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UP group default qlen 1000
link/ether fa:16:3e:47:49:52 brd ff:ff:ff:ff:ff:ff
inet 50.50.220.196/24 scope global eth0.37
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe47:4952/64 scope link
valid_lft forever preferred_lft forever
[cloud-user@tempest-testnfvoffload-server-1664025114 ~]$ lspci | grep Mella
04:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
[cloud-user@tempest-testnfvoffload-server-630963766 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc mq state UP group default qlen 1000
link/ether fa:16:3e:aa:91:a9 brd ff:ff:ff:ff:ff:ff
altname enp4s0
inet 20.20.220.152/24 brd 20.20.220.255 scope global dynamic noprefixroute eth0
valid_lft 85778sec preferred_lft 85778sec
inet6 fe80::f816:3eff:feaa:91a9/64 scope link
valid_lft forever preferred_lft forever
3: eth0.37@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UP group default qlen 1000
link/ether fa:16:3e:34:ed:d2 brd ff:ff:ff:ff:ff:ff
inet 50.50.220.175/24 scope global eth0.37
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe34:edd2/64 scope link
valid_lft forever preferred_lft forever
IPERF COMMANDS
iperf3 -s -B 50.50.220.196 -p 8645
iperf3 -c 50.50.220.196 -T s2 -p 8645 -t 100
IPERF OUTPUT
sender
s2: [ 5] 96.00-97.00 sec 1.15 GBytes 9.84 Gbits/sec 0 4.35 MBytes
s2: [ 5] 97.00-98.00 sec 1.14 GBytes 9.82 Gbits/sec 0 4.35 MBytes
s2: [ 5] 98.00-99.00 sec 1.14 GBytes 9.83 Gbits/sec 0 4.35 MBytes
s2: [ 5] 98.00-99.00 sec 1.14 GBytes 9.83 Gbits/sec 0 4.35 MBytes
s2: - - - - - - - - - - - - - - - - - - - - - - - - -
s2: [ ID] Interval Transfer Bitrate Retr
s2: [ 5] 0.00-99.00 sec 114 GBytes 9.88 Gbits/sec 0 sender
s2: [ 5] 0.00-99.00 sec 0.00 Bytes 0.00 bits/sec receiver
receiver
[ 5] 93.00-94.00 sec 1.14 GBytes 9.78 Gbits/sec
[ 5] 94.00-95.00 sec 1.13 GBytes 9.71 Gbits/sec
[ 5] 95.00-96.00 sec 1.13 GBytes 9.74 Gbits/sec
[ 5] 96.00-97.00 sec 1.13 GBytes 9.72 Gbits/sec
[ 5] 97.00-98.00 sec 1.14 GBytes 9.75 Gbits/sec
[ 5] 98.00-99.00 sec 1.14 GBytes 9.80 Gbits/sec
[ 5] 99.00-99.56 sec 660 MBytes 9.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-99.56 sec 113 GBytes 9.76 Gbits/sec receiver
FLOWS
compute0
ufid:ecb49555-034b-437d-821d-0161d0c72ba7, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_8),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:a9:d9:59,dst=fa:16:3e:c6:bd:1b),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no), packets:732, bytes:90322, used:2.730s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0xca71,src=192.168.32.171,dst=192.168.32.142,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789
ufid:f7a3d77f-389c-4efa-a73f-3fd425d04372, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_8),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:47:49:52,dst=fa:16:3e:34:ed:d2),eth_type(0x8100),vlan(vid=37,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no)), packets:14705051, bytes:123244426653, used:0.680s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0x7c2,src=192.168.32.171,dst=192.168.32.175,ttl=64,tp_dst=4789,flags(key))),pop_vlan,vxlan_sys_4789
ufid:692511bc-e579-4a9f-952d-15fe05094edf, skb_priority(0/0),tunnel(tun_id=0x5bc1,src=192.168.32.142,dst=192.168.32.171,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1411, bytes:76194, used:0.680s, offloaded:yes, dp:tc, actions:drop
ufid:84521901-b921-4104-9180-49d45747c6d7, skb_priority(0/0),tunnel(tun_id=0xca71,src=192.168.32.142,dst=192.168.32.171,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:c6:bd:1b,dst=fa:16:3e:a9:d9:59),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:721, bytes:88899, used:2.730s, offloaded:yes, dp:tc, actions:ens2f0np0_8
ufid:7fd4b53b-0265-4ed4-a5a8-ec9ac1c8ff2b, skb_priority(0/0),tunnel(tun_id=0x7c2,src=192.168.32.175,dst=192.168.32.171,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:34:ed:d2,dst=fa:16:3e:47:49:52),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:14532606, bytes:122547326619, used:0.680s, offloaded:yes, dp:tc, actions:push_vlan(vid=37,pcp=0),ens2f0np0_8
compute1
ufid:98d26dfb-abbc-4b43-bc9a-a7180d4e570a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:aa:91:a9,dst=fa:16:3e:c6:bd:1b),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no), packets:712, bytes:106391, used:5.380s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0xca71,src=192.168.32.175,dst=192.168.32.142,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789
ufid:f8bb9a4b-be84-4059-80b1-6a1e560afd9f, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:34:ed:d2,dst=fa:16:3e:47:49:52),eth_type(0x8100),vlan(vid=37,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no)), packets:14439745, bytes:121714575697, used:1.290s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0x7c2,src=192.168.32.175,dst=192.168.32.171,ttl=64,tp_dst=4789,flags(key))),pop_vlan,vxlan_sys_4789
ufid:11b9081a-16f1-4e36-b3d6-18d123815eec, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:aa:91:a9,dst=fa:16:3e:c6:bd:1b),eth_type(0x0806),arp(sip=0.0.0.0/0.0.0.0,tip=0.0.0.0/0.0.0.0,op=0/0,sha=00:00:00:00:00:00/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00), packets:0, bytes:0, used:never, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0xca71,src=192.168.32.175,dst=192.168.32.142,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789
ufid:a62a9579-51bf-4675-9f53-c556bf0176f1, skb_priority(0/0),tunnel(tun_id=0xca71,src=192.168.32.142,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:c6:bd:1b,dst=fa:16:3e:aa:91:a9),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:687, bytes:84751, used:5.380s, offloaded:yes, dp:tc, actions:ens2f0np0_7
ufid:83dfb631-c1d3-4175-8d6d-73ec363bbabd, skb_priority(0/0),tunnel(tun_id=0x7c2,src=192.168.32.171,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:47:49:52,dst=fa:16:3e:34:ed:d2),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:14699815, bytes:123244060272, used:1.290s, offloaded:yes, dp:tc, actions:push_vlan(vid=37,pcp=0),ens2f0np0_7
ufid:e9a4172b-7845-47e9-9799-4a602bca8669, skb_priority(0/0),tunnel(tun_id=0x5bc1,src=192.168.32.142,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1398, bytes:75492, used:2.310s, offloaded:yes, dp:tc, actions:drop
ufid:2128d151-8924-4598-b9ce-fac5ca14db0d, skb_priority(0/0),tunnel(tun_id=0xca71,src=192.168.32.142,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:c6:bd:1b,dst=fa:16:3e:aa:91:a9),eth_type(0x0806),arp(sip=0.0.0.0/0.0.0.0,tip=0.0.0.0/0.0.0.0,op=0/0,sha=00:00:00:00:00:00/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00), packets:0, bytes:0, used:never, offloaded:yes, dp:tc, actions:ens2f0np0_7
TCPDUMP REPRESENTOR PORT
compute0
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens2f0np0_8, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:23:45.172228 fa:16:3e:47:49:52 > fa:16:3e:34:ed:d2, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.196.43572 > 50.50.220.175.8645: Flags [S], seq 3002515508, win 26730, options [mss 8910,sackOK,TS val 3188319020 ecr 0,nop,wscale 7], length 0
10:23:45.175458 fa:16:3e:34:ed:d2 > fa:16:3e:47:49:52, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.175.8645 > 50.50.220.196.43572: Flags [S.], seq 2150371723, ack 3002515509, win 26694, options [mss 8910,sackOK,TS val 2321776299 ecr 3188319020,nop,wscale 7], length 0
2 packets captured
2 packets received by filter
0 packets dropped by kernel
compute1
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens2f0np0_7, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:23:45.173741 fa:16:3e:47:49:52 > fa:16:3e:34:ed:d2, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.196.43572 > 50.50.220.175.8645: Flags [S], seq 3002515508, win 26730, options [mss 8910,sackOK,TS val 3188319020 ecr 0,nop,wscale 7], length 0
10:23:45.173795 fa:16:3e:34:ed:d2 > fa:16:3e:47:49:52, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.175.8645 > 50.50.220.196.43572: Flags [S.], seq 2150371723, ack 3002515509, win 26694, options [mss 8910,sackOK,TS val 2321776299 ecr 3188319020,nop,wscale 7], length 0
2 packets captured
2 packets received by filter
0 packets dropped by kernel
COMPUTE NICS
[tripleo-admin@compute-0 ~]$ lspci | grep Mellanox
17:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
17:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
17:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01)
17:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:00.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:01.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:01.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:01.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:01.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:01.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:04.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:04.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:04.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:04.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:04.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:05.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:05.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:05.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:05.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
17:05.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
|
Description of problem: VXLAN OVS HW-Offload trunk not working. Parent VXLAN port is offloaded. Subport VXLAN port is not offloaded. This ticket was opened because OVS uses internal OVS bridges interfaces on the OVS flows when there is VxLAN network child port on a Neutron trunk, because internal interfaces cannot be offloaded via TC (no netdev created for those internal interfaces) there is no way to create efficient SR-IOV port with neutron trunk using VxLAN networks as child ports because the flows don't get offload to the NIC. Here are the scenarios where HW-Offload works: - VLAN Interface: Yes - VXLAN Interface: Yes - Trunk with VLAN parent port and VLAN subport: Yes Scenario where HW-Offload does not work: - Trunk with VXLAN parent port and VXLAN subport: No ************************************* Vxlan Parent port traffic is offloaded: ************************************* [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo ovs-appctl dpctl/dump-flows --names -m | grep ens3f1_63 ufid:44d4af15-a4b7-41a9-9544-1c7ed36409c2, skb_priority(0/0),tunnel(tun_id=0x1,src=172.20.16.13,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:dd:c1:15,dst=fa:16:3e:23:ce:93),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:440303, bytes:29521266, used:0.820s, offloaded:yes, dp:tc, actions:ens3f1_63 ufid:d82d5dcb-81a6-4628-b85e-69232e5ff85d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens3f1_63),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:23:ce:93,dst=fa:16:3e:dd:c1:15),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no), packets:4403866, bytes:37677035972, used:0.820s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0x1,src=172.20.16.9,dst=172.20.16.13,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789 [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo tc -s filter show dev ens3f1_63 ingress filter protocol ip pref 5 flower chain 0 filter protocol ip pref 5 flower chain 0 handle 0x1 dst_mac fa:16:3e:dd:c1:15 src_mac fa:16:3e:23:ce:93 eth_type ipv4 ip_tos 0/0x3 ip_flags nofrag in_hw in_hw_count 1 action order 1: tunnel_key set src_ip 172.20.16.9 dst_ip 172.20.16.13 key_id 1 dst_port 4789 nocsum ttl 64 pipe index 4 ref 1 bind 1 installed 20 sec used 20 sec firstused 20 sec Action statistics: Sent 59588956 bytes 7047 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device vxlan_sys_4789) stolen index 11 ref 1 bind 1 installed 20 sec used 1 sec firstused 20 sec Action statistics: Sent 45126426412 bytes 5274813 pkt (dropped 0, overlimits 0 requeues 0) Sent software 59588956 bytes 7047 pkt Sent hardware 45066837456 bytes 5267766 pkt backlog 0b 0p requeues 0 cookie cb5d2dd82846a68123695eb85df85f2e no_percpu used_hw_stats delayed [heat-admin@tpa-vim-p-computecl-1 ~]$ ******************************************************************************** Vxlan subport traffic NOT offloaded: Engress offloaded but Egress NOT offloaded ******************************************************************************** [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo ovs-appctl dpctl/dump-flows --names -m | grep ens3f1_63 ufid:2fbbb4ef-1d81-4ba5-850d-39a9b869b393, skb_priority(0/0),tunnel(tun_id=0x5,src=172.20.16.13,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:fa:be:a1,dst=fa:16:3e:23:ce:93),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:551122, bytes:36436928, used:1.700s, offloaded:yes, dp:tc, actions:push_vlan(vid=13,pcp=0),ens3f1_63 ufid:753ba6c2-8fac-4db1-971a-54039a819b56, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens3f1_63),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:23:ce:93,dst=fa:16:3e:fa:be:a1),eth_type(0x8100),vlan(vid=13,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no)), packets:4734768, bytes:40021087310, used:0.000s, dp:tc, actions:set(tunnel(tun_id=0x5,src=172.20.16.9,dst=172.20.16.13,ttl=64,tp_dst=4789,flags(key))),pop_vlan,vxlan_sys_4789 [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo tc -s filter show dev ens3f1_63 ingress filter protocol 802.1Q pref 4 flower chain 0 filter protocol 802.1Q pref 4 flower chain 0 handle 0x1 vlan_id 13 vlan_prio 0 vlan_ethtype ip dst_mac fa:16:3e:fa:be:a1 src_mac fa:16:3e:23:ce:93 eth_type ipv4 ip_tos 0/0x3 ip_flags nofrag not_in_hw action order 1: tunnel_key set src_ip 172.20.16.9 dst_ip 172.20.16.13 key_id 5 dst_port 4789 nocsum ttl 64 pipe index 1 ref 1 bind 1 installed 57 sec firstused 57 sec Action statistics: Sent 135283398847 bytes 16005737 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 no_percpu action order 2: vlan pop pipe index 6 ref 1 bind 1 installed 57 sec firstused 57 sec Action statistics: Sent 135283398847 bytes 16005737 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 no_percpu action order 3: mirred (Egress Redirect to device vxlan_sys_4789) stolen index 6 ref 1 bind 1 installed 57 sec firstused 57 sec Action statistics: Sent 135283398847 bytes 16005737 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 cookie e989e059be4910d1ddfb1ca7f73179f4 no_percpu [heat-admin@tpa-vim-p-computecl-1 ~]$ The value is set correctly on the host as you can see below: [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo mstconfig -d d8:00.0 -e q ESWITCH_IPV4_TTL_MODIFY_ENABLE Device #1: ---------- Device type: ConnectX5 Name: 0V5DG9_0TDNNT_Ax Description: ConnectX-5 EN network interface card; 25GbE Dual-port SFP28; SOCKET DIRECT ; PCIe3.0 2X8 Device: d8:00.0 Configurations: Default Current Next Boot ESWITCH_IPV4_TTL_MODIFY_ENABLE False(0) False(0) False(0) [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo devlink dev eswitch show pci/0000:d8:00.0 pci/0000:d8:00.0: mode switchdev inline-mode none encap-mode basic [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo ovs-appctl dpctl/dump-flows -m | grep 'offloaded:yes' ufid:0d30c164-ddea-4bac-b66c-e00d79f01796, skb_priority(0/0),tunnel(tun_id=0x2,src=172.20.16.11,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:68:61:3f,dst=fa:16:3e:23:ce:93),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:61456, bytes:6022688, used:0.470s, offloaded:yes, dp:tc, actions:push_vlan(vid=12,pcp=0),ens3f1_63 ufid:b8883ac7-2bd7-4673-acd2-8d79c97b83f0, skb_priority(0/0),tunnel(tun_id=0x2,src=172.20.16.11,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:68:61:3f,dst=fa:16:3e:23:ce:93),eth_type(0x0806),arp(sip=0.0.0.0/0.0.0.0,tip=0.0.0.0/0.0.0.0,op=0/0,sha=00:00:00:00:00:00/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00), packets:1, bytes:42, used:0.470s, offloaded:yes, dp:tc, actions:push_vlan(vid=12,pcp=0),ens3f1_63 [heat-admin@tpa-vim-p-computecl-1 ~]$ Version-Release number of selected component (if applicable): How reproducible: all the time