Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2040292

Summary:	[DPDK][Mellanox][NIC Partitioning] Port creation fails on CX4 VF with higher order of PCI address.
Product:	Red Hat OpenStack	Reporter:	Karthik Sundaravel <ksundara>
Component:	openvswitch	Assignee:	Karthik Sundaravel <ksundara>
Status:	CLOSED COMPLETED	QA Contact:	Eran Kuris <ekuris>
Severity:	medium	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	apevec, cfields, cfontain, chrisw, hakhande, mnietoji, supadhya
Target Milestone:	z6	Keywords:	Bugfix, OtherQA, Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2048601 (view as bug list)		Environment:
Last Closed:	2024-12-15 08:46:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2048601
Bug Blocks:

Description Karthik Sundaravel 2022-01-13 11:39:33 UTC

Description of problem:

The OSP deployment involves creation of 64 VFs on 4 CX4 ports (256 VFs), spread across 2 NUMA. A dpdk bond is created between VF0 of port 0 (NUMA0) and VF0 of port 3 (NUMA1). The port creation fails with below error.

    Bridge br-link0
        fail_mode: standalone
        datapath_type: netdev
        Port dpdkbond0
            Interface dpdk1
                type: dpdk
                options: {dpdk-devargs="0000:d8:00.2", n_rxq="1"}
                error: "Error attaching device '0000:d8:00.2' to DPDK"
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:12:00.2", n_rxq="1"}
        Port br-link0
            Interface br-link0
                type: internal
    ovs_version: "2.13.4"

How reproducible:

As a quick reproducer with limited VFs, we could build Openvswitch with CONFIG_RTE_MAX_ETHPORTS set to 32 and create a user bridge with dpdk bonds involving a VF with higher order PCI address.


Actual results:
Port creation fails

Expected results:
Port creation shall be successful.

SOS reports are attached in the case.
PS: The port creations were successful when the customer went ahead with 32 VFs per port.

Comment 1 Christophe Fontaine 2022-01-13 12:34:39 UTC

When OVS-DPDK starts, it will try to probe all available devices.
With MLX NICs, because of the bifurcated driver, ovs will probe all the VFs, and hits the limit (RTE_MAX_ETHPORTS).

RTE_MAX_ETHPORTS was set to 32 in the log below, but is set to 128 in what we ship.


2022-01-06T08:11:48.252Z|00088|dpdk|INFO|EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:c4:04.0 (socket 0)
2022-01-06T08:11:48.281Z|00089|dpdk|ERR|mlx5_net: unable to allocate switch domain: Operation not supported
2022-01-06T08:11:48.281Z|00090|dpdk|ERR|mlx5_net: probe of PCI device 0000:c4:04.0 aborted after encountering an error: Operation not supported
2022-01-06T08:11:48.281Z|00091|dpdk|ERR|mlx5_common: Failed to load driver mlx5_eth
2022-01-06T08:11:48.282Z|00092|dpdk|ERR|EAL: Requested device 0000:c4:04.0 cannot be used
[...]
2022-01-06T08:12:00.314Z|00748|dpdk|ERR|EAL: Driver cannot attach the device (0000:c4:04.0)
2022-01-06T08:12:00.314Z|00749|dpdk|ERR|EAL: Failed to attach device on primary process
2022-01-06T08:12:00.314Z|00750|netdev_dpdk|WARN|Error attaching device '0000:c4:04.0' to DPDK
2022-01-06T08:12:00.314Z|00751|netdev|WARN|dpdk0: could not set configuration (Invalid argument)
2022-01-06T08:12:00.314Z|00752|dpdk|ERR|Invalid port_id=32

As a workaround, we can give the allowlist/whitelist (-a or -w) parameter to the EAL, so we don't probe any device:
ovs-vsctl set open_vswitch . other_config:dpdk-extra="-a 0000:00:00.0"

Then, we can hot plug (aka  probe & attach ) the required VFs as needed.

Comment 10 Miguel Angel Nieto 2023-09-29 13:29:30 UTC

I still see the issue. Is anything missing?

There are 4 nics
18:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
18:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
3b:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)
3b:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)

Configure max num of vfs to 127
mstconfig  -d 18:00.0 s NUM_OF_VFS=127
mstconfig  -d 18:00.1 s NUM_OF_VFS=127
mstconfig  -d 3b:00.0 s NUM_OF_VFS=127
mstconfig  -d 3b:00.1 s NUM_OF_VFS=127


Configure 64 vfs per nic
echo 0 >/sys/class/net/ens6f0/device/sriov_numvfs
echo 0 >/sys/class/net/ens6f1/device/sriov_numvfs
echo 0 >/sys/class/net/ens1f0/device/sriov_numvfs
echo 0 >/sys/class/net/ens1f1/device/sriov_numvfs
echo 64 >/sys/class/net/ens6f0/device/sriov_numvfs
echo 64 >/sys/class/net/ens6f1/device/sriov_numvfs
echo 64 >/sys/class/net/ens1f0/device/sriov_numvfs
echo 64 >/sys/class/net/ens1f1/device/sriov_numvfs

[root@computehwoffload-r740-0 heat-admin]# lspci | grep Virtual | wc -l
256

18:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:00.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:01.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:02.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:03.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:04.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:05.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:06.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:07.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:08.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:08.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:10.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:11.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:12.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:13.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:14.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:15.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:16.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.3 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.4 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.5 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.6 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:17.7 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
18:18.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
3b:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:00.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:01.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:02.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:03.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:04.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:05.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:06.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:07.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:08.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:08.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:08.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:10.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:10.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:10.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:10.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:10.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:10.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:11.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:12.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:13.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:14.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:15.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:16.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:17.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:18.0 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)
3b:18.1 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)


Choose 2 vfs (with pci address in the end of the list)
[root@computehwoffload-r740-0 heat-admin]# ethtool -i ens1f1v50
driver: mlx5_core
version: 5.0-0
firmware-version: 24.35.2000 (MT_0000000765)
expansion-rom-version: 
bus-info: 0000:3b:16.4
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
[root@computehwoffload-r740-0 heat-admin]# ethtool -i ens1f0v50
driver: mlx5_core
version: 5.0-0
firmware-version: 24.35.2000 (MT_0000000765)
expansion-rom-version: 
bus-info: 0000:3b:06.5
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Configure ovs bond
driverctl set-override 0000:3b:16.4 vfio-pci
driverctl set-override 0000:3b:06.5 vfio-pci

[root@computehwoffload-r740-0 heat-admin]# driverctl list-overrides
0000:3b:06.5 vfio-pci
0000:3b:16.4 vfio-pci

ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk options:dpdk-devargs=0000:3b:16.4
ovs-vsctl add-port br0 dpdk-p1 -- set Interface dpdk-p1 type=dpdk options:dpdk-devargs=0000:3b:06.5

I still see the issue
    Bridge br0
        datapath_type: netdev
        Port dpdk-p0
            Interface dpdk-p0
                type: dpdk
                options: {dpdk-devargs="0000:3b:16.4"}
                error: "Error attaching device '0000:3b:16.4' to DPDK"
        Port br0
            Interface br0
                type: internal
        Port dpdk-p1
            Interface dpdk-p1
                type: dpdk
                options: {dpdk-devargs="0000:3b:06.5"}
                error: "Error attaching device '0000:3b:06.5' to DPDK"

Comment 11 Miguel Angel Nieto 2023-10-02 08:05:14 UTC

ok, i didnt set driver properly

driverctl set-override 0000:3b:16.4 mlx5_core
driverctl set-override 0000:3b:06.5 mlx5_core

driverctl list-overrides
0000:3b:06.5 mlx5_core
0000:3b:16.4 mlx5_core


ovs-vsctl add-bond br0 dpdkbond p0 p1 \
    -- set Interface p0 type=dpdk options:dpdk-devargs=0000:3b:16.4 \
    -- set Interface p1 type=dpdk options:dpdk-devargs=0000:3b:06.5


    Bridge br0
        datapath_type: netdev
        Port dpdkbond
            Interface p1
                type: dpdk
                options: {dpdk-devargs="0000:3b:06.5"}
            Interface p0
                type: dpdk
                options: {dpdk-devargs="0000:3b:16.4"}

Trying with other different vfs (first and last in the list)
driverctl set-override 0000:18:00.2 mlx5_core
driverctl set-override 0000:3b:18.1 mlx5_core

driverctl list-overrides
0000:18:00.2 mlx5_core
0000:3b:06.5 mlx5_core
0000:3b:16.4 mlx5_core
0000:3b:18.1 mlx5_core


    Bridge br0
        datapath_type: netdev
        Port dpdkbond
            Interface p1
                type: dpdk
                options: {dpdk-devargs="0000:3b:18.1"}
            Interface p0
                type: dpdk
                options: {dpdk-devargs="0000:18:00.2"}
        Port br0
            Interface br0
                type: internal
    ovs_version: "2.15.8"

ovs-vsctl del-port dpdkbond
ovs-vsctl add-bond br0 dpdkbond p0 p1 \
    -- set Interface p0 type=dpdk options:dpdk-devargs=0000:18:00.2 \
    -- set Interface p1 type=dpdk options:dpdk-devargs=0000:3b:18.1

So it looks like the issue is solved

Versions:
openvswitch2.15-2.15.0-136.el8fdp.x86_64
RHOS-16.2-RHEL-8-20230926.n.1