Description of problem: VXLAN OVS HW-Offload trunk not working. Parent VXLAN port is offloaded. Subport VXLAN port is not offloaded. This ticket was opened because OVS uses internal OVS bridges interfaces on the OVS flows when there is VxLAN network child port on a Neutron trunk, because internal interfaces cannot be offloaded via TC (no netdev created for those internal interfaces) there is no way to create efficient SR-IOV port with neutron trunk using VxLAN networks as child ports because the flows don't get offload to the NIC. Here are the scenarios where HW-Offload works: - VLAN Interface: Yes - VXLAN Interface: Yes - Trunk with VLAN parent port and VLAN subport: Yes Scenario where HW-Offload does not work: - Trunk with VXLAN parent port and VXLAN subport: No ************************************* Vxlan Parent port traffic is offloaded: ************************************* [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo ovs-appctl dpctl/dump-flows --names -m | grep ens3f1_63 ufid:44d4af15-a4b7-41a9-9544-1c7ed36409c2, skb_priority(0/0),tunnel(tun_id=0x1,src=172.20.16.13,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:dd:c1:15,dst=fa:16:3e:23:ce:93),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:440303, bytes:29521266, used:0.820s, offloaded:yes, dp:tc, actions:ens3f1_63 ufid:d82d5dcb-81a6-4628-b85e-69232e5ff85d, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens3f1_63),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:23:ce:93,dst=fa:16:3e:dd:c1:15),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no), packets:4403866, bytes:37677035972, used:0.820s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0x1,src=172.20.16.9,dst=172.20.16.13,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789 [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo tc -s filter show dev ens3f1_63 ingress filter protocol ip pref 5 flower chain 0 filter protocol ip pref 5 flower chain 0 handle 0x1 dst_mac fa:16:3e:dd:c1:15 src_mac fa:16:3e:23:ce:93 eth_type ipv4 ip_tos 0/0x3 ip_flags nofrag in_hw in_hw_count 1 action order 1: tunnel_key set src_ip 172.20.16.9 dst_ip 172.20.16.13 key_id 1 dst_port 4789 nocsum ttl 64 pipe index 4 ref 1 bind 1 installed 20 sec used 20 sec firstused 20 sec Action statistics: Sent 59588956 bytes 7047 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 no_percpu used_hw_stats delayed action order 2: mirred (Egress Redirect to device vxlan_sys_4789) stolen index 11 ref 1 bind 1 installed 20 sec used 1 sec firstused 20 sec Action statistics: Sent 45126426412 bytes 5274813 pkt (dropped 0, overlimits 0 requeues 0) Sent software 59588956 bytes 7047 pkt Sent hardware 45066837456 bytes 5267766 pkt backlog 0b 0p requeues 0 cookie cb5d2dd82846a68123695eb85df85f2e no_percpu used_hw_stats delayed [heat-admin@tpa-vim-p-computecl-1 ~]$ ******************************************************************************** Vxlan subport traffic NOT offloaded: Engress offloaded but Egress NOT offloaded ******************************************************************************** [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo ovs-appctl dpctl/dump-flows --names -m | grep ens3f1_63 ufid:2fbbb4ef-1d81-4ba5-850d-39a9b869b393, skb_priority(0/0),tunnel(tun_id=0x5,src=172.20.16.13,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:fa:be:a1,dst=fa:16:3e:23:ce:93),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:551122, bytes:36436928, used:1.700s, offloaded:yes, dp:tc, actions:push_vlan(vid=13,pcp=0),ens3f1_63 ufid:753ba6c2-8fac-4db1-971a-54039a819b56, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens3f1_63),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:23:ce:93,dst=fa:16:3e:fa:be:a1),eth_type(0x8100),vlan(vid=13,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no)), packets:4734768, bytes:40021087310, used:0.000s, dp:tc, actions:set(tunnel(tun_id=0x5,src=172.20.16.9,dst=172.20.16.13,ttl=64,tp_dst=4789,flags(key))),pop_vlan,vxlan_sys_4789 [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo tc -s filter show dev ens3f1_63 ingress filter protocol 802.1Q pref 4 flower chain 0 filter protocol 802.1Q pref 4 flower chain 0 handle 0x1 vlan_id 13 vlan_prio 0 vlan_ethtype ip dst_mac fa:16:3e:fa:be:a1 src_mac fa:16:3e:23:ce:93 eth_type ipv4 ip_tos 0/0x3 ip_flags nofrag not_in_hw action order 1: tunnel_key set src_ip 172.20.16.9 dst_ip 172.20.16.13 key_id 5 dst_port 4789 nocsum ttl 64 pipe index 1 ref 1 bind 1 installed 57 sec firstused 57 sec Action statistics: Sent 135283398847 bytes 16005737 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 no_percpu action order 2: vlan pop pipe index 6 ref 1 bind 1 installed 57 sec firstused 57 sec Action statistics: Sent 135283398847 bytes 16005737 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 no_percpu action order 3: mirred (Egress Redirect to device vxlan_sys_4789) stolen index 6 ref 1 bind 1 installed 57 sec firstused 57 sec Action statistics: Sent 135283398847 bytes 16005737 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 cookie e989e059be4910d1ddfb1ca7f73179f4 no_percpu [heat-admin@tpa-vim-p-computecl-1 ~]$ The value is set correctly on the host as you can see below: [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo mstconfig -d d8:00.0 -e q ESWITCH_IPV4_TTL_MODIFY_ENABLE Device #1: ---------- Device type: ConnectX5 Name: 0V5DG9_0TDNNT_Ax Description: ConnectX-5 EN network interface card; 25GbE Dual-port SFP28; SOCKET DIRECT ; PCIe3.0 2X8 Device: d8:00.0 Configurations: Default Current Next Boot ESWITCH_IPV4_TTL_MODIFY_ENABLE False(0) False(0) False(0) [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo devlink dev eswitch show pci/0000:d8:00.0 pci/0000:d8:00.0: mode switchdev inline-mode none encap-mode basic [heat-admin@tpa-vim-p-computecl-1 ~]$ sudo ovs-appctl dpctl/dump-flows -m | grep 'offloaded:yes' ufid:0d30c164-ddea-4bac-b66c-e00d79f01796, skb_priority(0/0),tunnel(tun_id=0x2,src=172.20.16.11,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:68:61:3f,dst=fa:16:3e:23:ce:93),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:61456, bytes:6022688, used:0.470s, offloaded:yes, dp:tc, actions:push_vlan(vid=12,pcp=0),ens3f1_63 ufid:b8883ac7-2bd7-4673-acd2-8d79c97b83f0, skb_priority(0/0),tunnel(tun_id=0x2,src=172.20.16.11,dst=172.20.16.9,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:68:61:3f,dst=fa:16:3e:23:ce:93),eth_type(0x0806),arp(sip=0.0.0.0/0.0.0.0,tip=0.0.0.0/0.0.0.0,op=0/0,sha=00:00:00:00:00:00/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00), packets:1, bytes:42, used:0.470s, offloaded:yes, dp:tc, actions:push_vlan(vid=12,pcp=0),ens3f1_63 [heat-admin@tpa-vim-p-computecl-1 ~]$ Version-Release number of selected component (if applicable): How reproducible: all the time
Hi, I was able to reproduce the issue with Connect X5 And I have tested that with Connect X6 the issue is NOT reproduced Used puddle RHOS-17.1-RHEL-9-20230628.n.2 Testcase run sucessfully when configured vxlan in parent and child network python -m testtools.run nfv_tempest_plugin.tests.scenario.test_nfv_offload.TestNfvOffload.test_offload_tcp_trunk_vlan Ran 1 test in 2411.853s OK NETWORKS (overcloud) [stack@undercloud-0 ~]$ openstack network show 354e8807-87f6-40d0-99c4-e3cc1e3f3648 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2023-07-03T09:45:46Z | | description | | | dns_domain | None | | id | 354e8807-87f6-40d0-99c4-e3cc1e3f3648 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 8950 | | name | mellanox-vxlan-provider | | port_security_enabled | False | | project_id | 994c523a6b724c3298fec775380f4c6b | | provider:network_type | vxlan | | provider:physical_network | None | | provider:segmentation_id | 51825 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | d959a665-2e69-43ca-a613-bea1b6d6a5d4 | | tags | | | updated_at | 2023-07-03T09:45:47Z | +---------------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack network show 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2023-07-03T09:45:47Z | | description | | | dns_domain | None | | id | 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 8492 | | name | trunk_network | | port_security_enabled | False | | project_id | 994c523a6b724c3298fec775380f4c6b | | provider:network_type | vxlan | | provider:physical_network | None | | provider:segmentation_id | 1986 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 7a957ce6-4dc0-4295-a1a5-5ec968b21fbf | | tags | | | updated_at | 2023-07-03T09:45:47Z | +---------------------------+--------------------------------------+ SERVERS (overcloud) [stack@undercloud-0 ~]$ openstack server list --all-projects +--------------------------------------+------------------------------------------+--------+------------------------------------------------------+----------------------------------------------+--------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------------------------------------------+--------+------------------------------------------------------+----------------------------------------------+--------------------+ | c459d5e1-af83-4eae-8e29-f8f23155eb52 | tempest-TestNfvOffload-server-1664025114 | ACTIVE | mellanox-vxlan-provider=10.46.228.106, 20.20.220.198 | rhel-guest-image-nfv-2-8.7-1660.x86_64.qcow2 | nfv_qe_ag_flavor_1 | | 0e611841-dc65-47a1-85b1-ee874f586079 | tempest-TestNfvOffload-server-630963766 | ACTIVE | mellanox-vxlan-provider=10.46.228.100, 20.20.220.152 | rhel-guest-image-nfv-2-8.7-1660.x86_64.qcow2 | nfv_qe_ag_flavor_0 | +--------------------------------------+------------------------------------------+--------+------------------------------------------------------+----------------------------------------------+-------------------- + PORTS (overcloud) [stack@undercloud-0 ~]$ openstack port show tempest-port-smoke-1748868985 +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | compute-1.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:17:01.3', pci_vendor_info='15b3:101e', physical_network= | | binding_vif_details | bridge_name='tbr-2e0fb415-f', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-03T09:45:48Z | | data_plane_status | None | | description | | | device_id | c459d5e1-af83-4eae-8e29-f8f23155eb52 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='20.20.220.198', subnet_id='d959a665-2e69-43ca-a613-bea1b6d6a5d4' | | id | 06136fd2-1b58-4d32-baac-f0a4fd88f7dc | | ip_allocation | None | | mac_address | fa:16:3e:a9:d9:59 | | name | tempest-port-smoke-1748868985 | | network_id | 354e8807-87f6-40d0-99c4-e3cc1e3f3648 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 994c523a6b724c3298fec775380f4c6b | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | {'trunk_id': '2e0fb415-fd29-4388-8bd8-208c8a8016ad', 'sub_ports': [{'segmentation_id': 37, 'segmentation_type': 'vlan', 'port_id': '39fb968b-09bc-4f65-8b02-0121ec89b1fd'}]} | | updated_at | 2023-07-03T09:46:11Z | +-------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port show d8a42b85-fa38-4de6-8de9-672fd3f4b913 --fit-width +-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | compute-0.localdomain | | binding_profile | capabilities='['switchdev']', pci_slot='0000:17:01.2', pci_vendor_info='15b3:101e', physical_network= | | binding_vif_details | bridge_name='tbr-7685e174-f', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | direct | | created_at | 2023-07-03T09:45:47Z | | data_plane_status | None | | description | | | device_id | 0e611841-dc65-47a1-85b1-ee874f586079 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='20.20.220.152', subnet_id='d959a665-2e69-43ca-a613-bea1b6d6a5d4' | | id | d8a42b85-fa38-4de6-8de9-672fd3f4b913 | | ip_allocation | None | | mac_address | fa:16:3e:aa:91:a9 | | name | tempest-port-smoke-1511182731 | | network_id | 354e8807-87f6-40d0-99c4-e3cc1e3f3648 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 994c523a6b724c3298fec775380f4c6b | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | {'trunk_id': '7685e174-f1b1-4305-b372-6acfb2688460', 'sub_ports': [{'segmentation_id': 37, 'segmentation_type': 'vlan', 'port_id': 'cd69a024-6865-4859-b2e6-91b423649474', | | | 'mac_address': 'fa:16:3e:34:ed:d2'}]} | | updated_at | 2023-07-03T09:46:00Z | +-------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port show 39fb968b-09bc-4f65-8b02-0121ec89b1fd +-------------------------+--------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | compute-1.localdomain | | binding_profile | | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2023-07-03T09:45:49Z | | data_plane_status | None | | description | | | device_id | | | device_owner | trunk:subport | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='50.50.220.196', subnet_id='7a957ce6-4dc0-4295-a1a5-5ec968b21fbf' | | id | 39fb968b-09bc-4f65-8b02-0121ec89b1fd | | ip_allocation | None | | mac_address | fa:16:3e:47:49:52 | | name | tempest-port-smoke-15951452 | | network_id | 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 994c523a6b724c3298fec775380f4c6b | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-03T09:46:11Z | +-------------------------+--------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack port show cd69a024-6865-4859-b2e6-91b423649474 +-------------------------+--------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | compute-0.localdomain | | binding_profile | | | binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='False', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2023-07-03T09:45:48Z | | data_plane_status | None | | description | | | device_id | | | device_owner | trunk:subport | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='50.50.220.175', subnet_id='7a957ce6-4dc0-4295-a1a5-5ec968b21fbf' | | id | cd69a024-6865-4859-b2e6-91b423649474 | | ip_allocation | None | | mac_address | fa:16:3e:34:ed:d2 | | name | tempest-port-smoke-1567555246 | | network_id | 7e8cf9cb-d204-4d2b-a62b-20a07bac69c1 | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | 994c523a6b724c3298fec775380f4c6b | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 4 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2023-07-03T09:46:00Z | +-------------------------+--------------------------------------------------------------------------------------------------------------+ TRUNK (overcloud) [stack@undercloud-0 ~]$ openstack network trunk show 2e0fb415-fd29-4388-8bd8-208c8a8016ad +-----------------+------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | created_at | 2023-07-03T09:45:50Z | | description | | | id | 2e0fb415-fd29-4388-8bd8-208c8a8016ad | | name | | | port_id | 06136fd2-1b58-4d32-baac-f0a4fd88f7dc | | project_id | 6ccccae1eb15400c9c6d2b66fb344a63 | | revision_number | 3 | | status | ACTIVE | | sub_ports | port_id='39fb968b-09bc-4f65-8b02-0121ec89b1fd', segmentation_id='37', segmentation_type='vlan' | | tags | [] | | tenant_id | 6ccccae1eb15400c9c6d2b66fb344a63 | | updated_at | 2023-07-03T09:46:11Z | +-----------------+------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack network trunk show 7685e174-f1b1-4305-b372-6acfb2688460 +-----------------+------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | created_at | 2023-07-03T09:45:50Z | | description | | | id | 7685e174-f1b1-4305-b372-6acfb2688460 | | name | | | port_id | d8a42b85-fa38-4de6-8de9-672fd3f4b913 | | project_id | 6ccccae1eb15400c9c6d2b66fb344a63 | | revision_number | 3 | | status | ACTIVE | | sub_ports | port_id='cd69a024-6865-4859-b2e6-91b423649474', segmentation_id='37', segmentation_type='vlan' | | tags | [] | | tenant_id | 6ccccae1eb15400c9c6d2b66fb344a63 | | updated_at | 2023-07-03T09:46:00Z | +-----------------+------------------------------------------------------------------------------------------------+ VMS [cloud-user@tempest-testnfvoffload-server-1664025114 ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:a9:d9:59 brd ff:ff:ff:ff:ff:ff altname enp4s0 inet 20.20.220.198/24 brd 20.20.220.255 scope global dynamic noprefixroute eth0 valid_lft 85794sec preferred_lft 85794sec inet6 fe80::f816:3eff:fea9:d959/64 scope link valid_lft forever preferred_lft forever 3: eth0.37@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:47:49:52 brd ff:ff:ff:ff:ff:ff inet 50.50.220.196/24 scope global eth0.37 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe47:4952/64 scope link valid_lft forever preferred_lft forever [cloud-user@tempest-testnfvoffload-server-1664025114 ~]$ lspci | grep Mella 04:00.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) [cloud-user@tempest-testnfvoffload-server-630963766 ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:aa:91:a9 brd ff:ff:ff:ff:ff:ff altname enp4s0 inet 20.20.220.152/24 brd 20.20.220.255 scope global dynamic noprefixroute eth0 valid_lft 85778sec preferred_lft 85778sec inet6 fe80::f816:3eff:feaa:91a9/64 scope link valid_lft forever preferred_lft forever 3: eth0.37@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:34:ed:d2 brd ff:ff:ff:ff:ff:ff inet 50.50.220.175/24 scope global eth0.37 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe34:edd2/64 scope link valid_lft forever preferred_lft forever IPERF COMMANDS iperf3 -s -B 50.50.220.196 -p 8645 iperf3 -c 50.50.220.196 -T s2 -p 8645 -t 100 IPERF OUTPUT sender s2: [ 5] 96.00-97.00 sec 1.15 GBytes 9.84 Gbits/sec 0 4.35 MBytes s2: [ 5] 97.00-98.00 sec 1.14 GBytes 9.82 Gbits/sec 0 4.35 MBytes s2: [ 5] 98.00-99.00 sec 1.14 GBytes 9.83 Gbits/sec 0 4.35 MBytes s2: [ 5] 98.00-99.00 sec 1.14 GBytes 9.83 Gbits/sec 0 4.35 MBytes s2: - - - - - - - - - - - - - - - - - - - - - - - - - s2: [ ID] Interval Transfer Bitrate Retr s2: [ 5] 0.00-99.00 sec 114 GBytes 9.88 Gbits/sec 0 sender s2: [ 5] 0.00-99.00 sec 0.00 Bytes 0.00 bits/sec receiver receiver [ 5] 93.00-94.00 sec 1.14 GBytes 9.78 Gbits/sec [ 5] 94.00-95.00 sec 1.13 GBytes 9.71 Gbits/sec [ 5] 95.00-96.00 sec 1.13 GBytes 9.74 Gbits/sec [ 5] 96.00-97.00 sec 1.13 GBytes 9.72 Gbits/sec [ 5] 97.00-98.00 sec 1.14 GBytes 9.75 Gbits/sec [ 5] 98.00-99.00 sec 1.14 GBytes 9.80 Gbits/sec [ 5] 99.00-99.56 sec 660 MBytes 9.84 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-99.56 sec 113 GBytes 9.76 Gbits/sec receiver FLOWS compute0 ufid:ecb49555-034b-437d-821d-0161d0c72ba7, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_8),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:a9:d9:59,dst=fa:16:3e:c6:bd:1b),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no), packets:732, bytes:90322, used:2.730s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0xca71,src=192.168.32.171,dst=192.168.32.142,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789 ufid:f7a3d77f-389c-4efa-a73f-3fd425d04372, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_8),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:47:49:52,dst=fa:16:3e:34:ed:d2),eth_type(0x8100),vlan(vid=37,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no)), packets:14705051, bytes:123244426653, used:0.680s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0x7c2,src=192.168.32.171,dst=192.168.32.175,ttl=64,tp_dst=4789,flags(key))),pop_vlan,vxlan_sys_4789 ufid:692511bc-e579-4a9f-952d-15fe05094edf, skb_priority(0/0),tunnel(tun_id=0x5bc1,src=192.168.32.142,dst=192.168.32.171,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1411, bytes:76194, used:0.680s, offloaded:yes, dp:tc, actions:drop ufid:84521901-b921-4104-9180-49d45747c6d7, skb_priority(0/0),tunnel(tun_id=0xca71,src=192.168.32.142,dst=192.168.32.171,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:c6:bd:1b,dst=fa:16:3e:a9:d9:59),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:721, bytes:88899, used:2.730s, offloaded:yes, dp:tc, actions:ens2f0np0_8 ufid:7fd4b53b-0265-4ed4-a5a8-ec9ac1c8ff2b, skb_priority(0/0),tunnel(tun_id=0x7c2,src=192.168.32.175,dst=192.168.32.171,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:34:ed:d2,dst=fa:16:3e:47:49:52),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:14532606, bytes:122547326619, used:0.680s, offloaded:yes, dp:tc, actions:push_vlan(vid=37,pcp=0),ens2f0np0_8 compute1 ufid:98d26dfb-abbc-4b43-bc9a-a7180d4e570a, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:aa:91:a9,dst=fa:16:3e:c6:bd:1b),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no), packets:712, bytes:106391, used:5.380s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0xca71,src=192.168.32.175,dst=192.168.32.142,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789 ufid:f8bb9a4b-be84-4059-80b1-6a1e560afd9f, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:34:ed:d2,dst=fa:16:3e:47:49:52),eth_type(0x8100),vlan(vid=37,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0x3,ttl=0/0,frag=no)), packets:14439745, bytes:121714575697, used:1.290s, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0x7c2,src=192.168.32.175,dst=192.168.32.171,ttl=64,tp_dst=4789,flags(key))),pop_vlan,vxlan_sys_4789 ufid:11b9081a-16f1-4e36-b3d6-18d123815eec, skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens2f0np0_7),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:aa:91:a9,dst=fa:16:3e:c6:bd:1b),eth_type(0x0806),arp(sip=0.0.0.0/0.0.0.0,tip=0.0.0.0/0.0.0.0,op=0/0,sha=00:00:00:00:00:00/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00), packets:0, bytes:0, used:never, offloaded:yes, dp:tc, actions:set(tunnel(tun_id=0xca71,src=192.168.32.175,dst=192.168.32.142,ttl=64,tp_dst=4789,flags(key))),vxlan_sys_4789 ufid:a62a9579-51bf-4675-9f53-c556bf0176f1, skb_priority(0/0),tunnel(tun_id=0xca71,src=192.168.32.142,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:c6:bd:1b,dst=fa:16:3e:aa:91:a9),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:687, bytes:84751, used:5.380s, offloaded:yes, dp:tc, actions:ens2f0np0_7 ufid:83dfb631-c1d3-4175-8d6d-73ec363bbabd, skb_priority(0/0),tunnel(tun_id=0x7c2,src=192.168.32.171,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:47:49:52,dst=fa:16:3e:34:ed:d2),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:14699815, bytes:123244060272, used:1.290s, offloaded:yes, dp:tc, actions:push_vlan(vid=37,pcp=0),ens2f0np0_7 ufid:e9a4172b-7845-47e9-9799-4a602bca8669, skb_priority(0/0),tunnel(tun_id=0x5bc1,src=192.168.32.142,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:1398, bytes:75492, used:2.310s, offloaded:yes, dp:tc, actions:drop ufid:2128d151-8924-4598-b9ce-fac5ca14db0d, skb_priority(0/0),tunnel(tun_id=0xca71,src=192.168.32.142,dst=192.168.32.175,ttl=0/0,tp_dst=4789,flags(+key)),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(vxlan_sys_4789),packet_type(ns=0/0,id=0/0),eth(src=fa:16:3e:c6:bd:1b,dst=fa:16:3e:aa:91:a9),eth_type(0x0806),arp(sip=0.0.0.0/0.0.0.0,tip=0.0.0.0/0.0.0.0,op=0/0,sha=00:00:00:00:00:00/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00), packets:0, bytes:0, used:never, offloaded:yes, dp:tc, actions:ens2f0np0_7 TCPDUMP REPRESENTOR PORT compute0 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ens2f0np0_8, link-type EN10MB (Ethernet), snapshot length 262144 bytes 10:23:45.172228 fa:16:3e:47:49:52 > fa:16:3e:34:ed:d2, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.196.43572 > 50.50.220.175.8645: Flags [S], seq 3002515508, win 26730, options [mss 8910,sackOK,TS val 3188319020 ecr 0,nop,wscale 7], length 0 10:23:45.175458 fa:16:3e:34:ed:d2 > fa:16:3e:47:49:52, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.175.8645 > 50.50.220.196.43572: Flags [S.], seq 2150371723, ack 3002515509, win 26694, options [mss 8910,sackOK,TS val 2321776299 ecr 3188319020,nop,wscale 7], length 0 2 packets captured 2 packets received by filter 0 packets dropped by kernel compute1 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ens2f0np0_7, link-type EN10MB (Ethernet), snapshot length 262144 bytes 10:23:45.173741 fa:16:3e:47:49:52 > fa:16:3e:34:ed:d2, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.196.43572 > 50.50.220.175.8645: Flags [S], seq 3002515508, win 26730, options [mss 8910,sackOK,TS val 3188319020 ecr 0,nop,wscale 7], length 0 10:23:45.173795 fa:16:3e:34:ed:d2 > fa:16:3e:47:49:52, ethertype 802.1Q (0x8100), length 78: vlan 37, p 0, ethertype IPv4 (0x0800), 50.50.220.175.8645 > 50.50.220.196.43572: Flags [S.], seq 2150371723, ack 3002515509, win 26694, options [mss 8910,sackOK,TS val 2321776299 ecr 3188319020,nop,wscale 7], length 0 2 packets captured 2 packets received by filter 0 packets dropped by kernel COMPUTE NICS [tripleo-admin@compute-0 ~]$ lspci | grep Mellanox 17:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01) 17:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01) 17:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01) 17:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:00.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:01.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:01.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:01.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:01.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:01.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:04.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:04.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:04.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:04.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:04.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:05.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:05.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:05.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:05.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01) 17:05.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)