Bug 2059758

+++ This bug was initially created as a clone of Bug #2048601 +++

+++ This bug was initially created as a clone of Bug #2040292 +++

Description of problem:

The OSP deployment involves creation of 64 VFs on 4 CX4 ports (256 VFs), spread across 2 NUMA. A dpdk bond is created between VF0 of port 0 (NUMA0) and VF0 of port 3 (NUMA1). The port creation fails with below error.

    Bridge br-link0
        fail_mode: standalone
        datapath_type: netdev
        Port dpdkbond0
            Interface dpdk1
                type: dpdk
                options: {dpdk-devargs="0000:d8:00.2", n_rxq="1"}
                error: "Error attaching device '0000:d8:00.2' to DPDK"
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:12:00.2", n_rxq="1"}
        Port br-link0
            Interface br-link0
                type: internal
    ovs_version: "2.13.4"

How reproducible:

As a quick reproducer with limited VFs, we could build Openvswitch with CONFIG_RTE_MAX_ETHPORTS set to 32 and create a user bridge with dpdk bonds involving a VF with higher order PCI address.


Actual results:
Port creation fails

Expected results:
Port creation shall be successful.

SOS reports are attached in the case.
PS: The port creations were successful when the customer went ahead with 32 VFs per port.

--- Additional comment from Christophe Fontaine on 2022-01-13 12:34:39 UTC ---

When OVS-DPDK starts, it will try to probe all available devices.
With MLX NICs, because of the bifurcated driver, ovs will probe all the VFs, and hits the limit (RTE_MAX_ETHPORTS).

RTE_MAX_ETHPORTS was set to 32 in the log below, but is set to 128 in what we ship.


2022-01-06T08:11:48.252Z|00088|dpdk|INFO|EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:c4:04.0 (socket 0)
2022-01-06T08:11:48.281Z|00089|dpdk|ERR|mlx5_net: unable to allocate switch domain: Operation not supported
2022-01-06T08:11:48.281Z|00090|dpdk|ERR|mlx5_net: probe of PCI device 0000:c4:04.0 aborted after encountering an error: Operation not supported
2022-01-06T08:11:48.281Z|00091|dpdk|ERR|mlx5_common: Failed to load driver mlx5_eth
2022-01-06T08:11:48.282Z|00092|dpdk|ERR|EAL: Requested device 0000:c4:04.0 cannot be used
[...]
2022-01-06T08:12:00.314Z|00748|dpdk|ERR|EAL: Driver cannot attach the device (0000:c4:04.0)
2022-01-06T08:12:00.314Z|00749|dpdk|ERR|EAL: Failed to attach device on primary process
2022-01-06T08:12:00.314Z|00750|netdev_dpdk|WARN|Error attaching device '0000:c4:04.0' to DPDK
2022-01-06T08:12:00.314Z|00751|netdev|WARN|dpdk0: could not set configuration (Invalid argument)
2022-01-06T08:12:00.314Z|00752|dpdk|ERR|Invalid port_id=32

As a workaround, we can give the allowlist/whitelist (-a or -w) parameter to the EAL, so we don't probe any device:
ovs-vsctl set open_vswitch . other_config:dpdk-extra="-a 0000:00:00.0"

Then, we can hot plug (aka  probe & attach ) the required VFs as needed.

--- Additional comment from Chris Fields on 2022-01-25 20:12:10 UTC ---

In order to write a KCS on this we need template examples.  I'll suggest a 'post-install.yaml' like this one [1] except with this scripted action (instead of the mcast options shown currently):

sudo ovs-vsctl set open_vswitch . other_config:dpdk-extra="-a 0000:00:00.0"

Thoughts?  

For the subsequent probe and attachment of VF's; what are the actions?  

CFields

[1] https://code.engineering.redhat.com/gerrit/gitweb?p=nfv-qe.git;a=blob;f=ospd-16.2-vxlan-dpdk-sriov-ctlplane-dataplane-bonding-lacp-igmp-hybrid/post-install.yaml;hb=bc775078f303ad6948a0acbe67f0c088d63370bc

--- Additional comment from Christophe Fontaine on 2022-01-26 14:57:27 UTC ---

Instead of a post-install.yaml, can't we use the variable "tripleo_ovs_dpdk_extra" ?

https://github.com/openstack/tripleo-ansible/blob/8165e3e94209bbe9ab52cae06de25a56a8b4c812/tripleo_ansible/roles/tripleo_ovs_dpdk/defaults/main.yml#L25

--- Additional comment from Haresh Khandelwal on 2022-01-31 15:18:08 UTC ---

We have discussed this bug internally and below is what we think.
- post-install comes very late in the deployment cycle and this allowlisting is needed much earlier.
- we have tripleo-ansible role with tripleo_ovs_dpdk_extra But dont have corresponding tripleo heat parameter.
- NFV team will own this bz and provide a fix for tripleo-heat-templates. 
- Clone this Bz against FDP product so ovs team can fix actual issue

Thanks

--- Additional comment from Anita Tragler on 2022-01-31 17:15:56 CET ---

This is needed for OSP 16.2 customers requiring to use NIC partitioning
This issue is generic and affects OVS_DPDK NICs COnnectX-4 conenctx-5 that support more than 32 VFs , 64 VFs or higher (28 VFs)
We need DPDK support for higher order PCI IDs.
Is it possible to fix it in DPDK 21.11 and  backport to 20.11?

--- Additional comment from David Marchand on 2022-02-03 14:55:00 CET ---

Strictly speaking, DPDK does not limit "higher order" PCI IDs and there is nothing to "fix" in DPDK.

The issue is how OVS lets DPDK scan all PCI devices, regardless of those devices being used in OVS later.
Because of this, a limit on the number of ports in DPDK (RTE_ETH_MAXPORTS) is reached, and no other mlx device can be initialised in DPDK.

- One option is to make RTE_ETH_MAXPORTS dynamic, but this breaks API/ABI with a huge rework in DPDK.

- One other option is to avoid the initial DPDK scan (this is what the workaround I provided to NFV team).
  DPDK then only probes PCI devices which OVS passes on port addition.
  One note though, this partially breaks support of devices like mlx4 which have multiple ports for a single PCI device because it breaks use of syntax 'class=eth,mac=$MAC' for them.
  This does not affect other devices we currently support, and we dropped mlx4 support not so long ago, so this is not a big problem.

- One last option, probably the best compromise, is to raise the number of accepted ports (RTE_ETH_MAXPORTS) in the embedded DPDK in OVS downstream packages.
  We must decide on a best value (from report, 1024 seems high enough, the max value possible being UINT16_MAX, i.e. 64k).
  This requires QE to do sanity checks with new config.

--- Additional comment from Chris Fields on 2022-02-15 15:16:22 CET ---

Haresh, do you have feedback on the options David presented?

--- Additional comment from Haresh Khandelwal on 2022-02-21 08:04:27 CET ---

(In reply to David Marchand from comment #2)
> Strictly speaking, DPDK does not limit "higher order" PCI IDs and there is
> nothing to "fix" in DPDK.
> 
> The issue is how OVS lets DPDK scan all PCI devices, regardless of those
> devices being used in OVS later.
> Because of this, a limit on the number of ports in DPDK (RTE_ETH_MAXPORTS)
> is reached, and no other mlx device can be initialised in DPDK.
> 
> - One option is to make RTE_ETH_MAXPORTS dynamic, but this breaks API/ABI
> with a huge rework in DPDK.
> 
> - One other option is to avoid the initial DPDK scan (this is what the
> workaround I provided to NFV team).
>   DPDK then only probes PCI devices which OVS passes on port addition.
>   One note though, this partially breaks support of devices like mlx4 which
> have multiple ports for a single PCI device because it breaks use of syntax
> 'class=eth,mac=$MAC' for them.
>   This does not affect other devices we currently support, and we dropped
> mlx4 support not so long ago, so this is not a big problem.
> 
> - One last option, probably the best compromise, is to raise the number of
> accepted ports (RTE_ETH_MAXPORTS) in the embedded DPDK in OVS downstream
> packages.
>   We must decide on a best value (from report, 1024 seems high enough, the
> max value possible being UINT16_MAX, i.e. 64k).
>   This requires QE to do sanity checks with new config.

Ack on this 3rd option David, we can have sanity/perf jobs runs and confirm any regression.

Thanks

--- Additional comment from Red Hat Bugzilla on 2022-02-22 06:43:34 CET ---

remove performed by PnT Account Manager <pnt-expunge>