Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1971564

Summary: [RHEL9.0-BETA] pyverbs-tests fail with errors in "tests.test_mlx5_flow.Mlx5MatcherTest" and "tests.test_mlx5_mkey.Mlx5MkeyTest" on certain MLX5 devices
Product: Red Hat Enterprise Linux 9 Reporter: Brian Chae <bchae>
Component: rdma-coreAssignee: Nobody <nobody>
Status: CLOSED WONTFIX QA Contact: Infiniband QE <infiniband-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0CC: hwkernel-mgr, rdma-dev-team
Target Milestone: betaFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-12-14 07:27:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brian Chae 2021-06-14 10:59:07 UTC 
Description of problem:

pyverbs-tests fail due to 1 error in "tests.test_mlx5_flow.Mlx5MatcherTest" and 5 errors in "tests.test_mlx5_mkey.Mlx5MkeyTest" when tested on MLX5 devices on hosts rdma-dev-19/20 pair as well as on rdma-qe-06/07 pair (mlx5 CIB).

test_tx_packet_reformat (tests.test_mlx5_flow.Mlx5MatcherTest) ... ERROR
test_raw_modify_lag_port (tests.test_mlx5_lag_affinity.LagPortTestCase) ... skipped 'Set LAG affinity is not supported on this device'
test_rc_modify_lag_port (tests.test_mlx5_lag_affinity.LagPortTestCase) ... skipped 'Set LAG affinity is not supported on this device'
test_ud_modify_lag_port (tests.test_mlx5_lag_affinity.LagPortTestCase) ... skipped 'Set LAG affinity is not supported on this device'
test_mkey_interleaved (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register an interleaved memory layout using this mkey and ... ERROR
test_mkey_interleaved_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, configure it with interleaved memory layout using the new ... ERROR
test_mkey_list (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register a memory layout using this mkey and then perform ... ERROR
test_mkey_list_bad_flow (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register a memory layout using this mkey and then try to ... ERROR
test_mkey_list_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, configure it with memory layout using the new API and ... ERROR
test_mkey_sig_crc (tests.test_mlx5_mkey.Mlx5MkeyTest)

The HCA type with this issue is for both IB and ROCE, in case of hosts, rdma-dev-19/20.


Version-Release number of selected component (if applicable):

DISTRO=RHEL-9.0.0-20210610.2

Red Hat Enterprise Linux release 9.0 Beta (Plow)

Linux rdma-dev-19.lab.bos.redhat.com 5.13.0-0.rc4.33.el9.x86_64 #1 SMP Wed Jun 2 19:15:08 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.13.0-0.rc4.33.el9.x86_64 root=/dev/mapper/rhel_rdma--dev--19-root ro console=tty0 rd_NO_PLYMOUTH intel_idle.max_cstate=0 intel_iommu=on iommu=on processor.max_cstate=0 resume=/dev/mapper/rhel_rdma--dev--19-swap rd.lvm.lv=rhel_rdma-dev-19/root rd.lvm.lv=rhel_rdma-dev-19/swap console=ttyS1,115200

rdma-core-34.0-4.el9.x86_64

linux-firmware-20210315-120.el9.noarch

Installed:
  python3-pyverbs-34.0-4.el9.x86_64                                             



How reproducible:

100%

Steps to Reproduce:

1. With the above build, on rdma-dev-19/20 or rdma-qe-06/07
2. clone the "rdma-core"
3. build the "rdma-core"
4. execute the pyverbs tests

    ./build/bin/run_tests.py -v --dev $HCA_ID

    <HCA_ID:	mlx5_2 [IB] | mlx5_bond_0 [ROCE]>

Actual results:

======================================================================
ERROR: test_tx_packet_reformat (tests.test_mlx5_flow.Mlx5MatcherTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_flow.py", line 44, in func_wrapper
    cmd_out = Mlx5Context.devx_general_cmd(ctx, cmd_in,
  File "mlx5dv.pyx", line 226, in pyverbs.providers.mlx5.mlx5dv.Mlx5Context.devx_general_cmd
pyverbs.pyverbs_error.PyverbsRDMAError: DevX general command failed. Errno: 93, Protocol not supported

======================================================================
ERROR: test_mkey_interleaved (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register an interleaved memory layout using this mkey and
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 394, in test_mkey_interleaved
    self.create_players(Mlx5MkeyResources,
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 115, in create_players
    self.client = resource(**self.dev_info, **resource_arg)
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 44, in __init__
    self.create_mkey()
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 61, in create_mkey
    raise ex
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 57, in create_mkey
    self.mkey = Mlx5Mkey(self.pd, self.mkey_create_flags, 3)
  File "mlx5dv_mkey.pyx", line 186, in pyverbs.providers.mlx5.mlx5dv_mkey.Mlx5Mkey.__init__
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to create mkey. Errno: 93, Protocol not supported

======================================================================
ERROR: test_mkey_interleaved_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, configure it with interleaved memory layout using the new
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 427, in test_mkey_interleaved_new_api
    self.create_players(Mlx5MkeyResources,
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 115, in create_players
    self.client = resource(**self.dev_info, **resource_arg)
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 44, in __init__
    self.create_mkey()
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 61, in create_mkey
    raise ex
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 57, in create_mkey
    self.mkey = Mlx5Mkey(self.pd, self.mkey_create_flags, 3)
  File "mlx5dv_mkey.pyx", line 186, in pyverbs.providers.mlx5.mlx5dv_mkey.Mlx5Mkey.__init__
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to create mkey. Errno: 93, Protocol not supported

======================================================================
ERROR: test_mkey_list (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register a memory layout using this mkey and then perform
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 405, in test_mkey_list
    self.create_players(Mlx5MkeyResources,
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 115, in create_players
    self.client = resource(**self.dev_info, **resource_arg)
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 44, in __init__
    self.create_mkey()
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 61, in create_mkey
    raise ex
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 57, in create_mkey
    self.mkey = Mlx5Mkey(self.pd, self.mkey_create_flags, 3)
  File "mlx5dv_mkey.pyx", line 186, in pyverbs.providers.mlx5.mlx5dv_mkey.Mlx5Mkey.__init__
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to create mkey. Errno: 93, Protocol not supported

======================================================================
ERROR: test_mkey_list_bad_flow (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register a memory layout using this mkey and then try to
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 439, in test_mkey_list_bad_flow
    self.create_players(Mlx5MkeyResources,
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 115, in create_players
    self.client = resource(**self.dev_info, **resource_arg)
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 44, in __init__
    self.create_mkey()
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 61, in create_mkey
    raise ex
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 57, in create_mkey
    self.mkey = Mlx5Mkey(self.pd, self.mkey_create_flags, 3)
  File "mlx5dv_mkey.pyx", line 186, in pyverbs.providers.mlx5.mlx5dv_mkey.Mlx5Mkey.__init__
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to create mkey. Errno: 93, Protocol not supported

======================================================================
ERROR: test_mkey_list_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, configure it with memory layout using the new API and
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 416, in test_mkey_list_new_api
    self.create_players(Mlx5MkeyResources,
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 115, in create_players
    self.client = resource(**self.dev_info, **resource_arg)
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 44, in __init__
    self.create_mkey()
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 61, in create_mkey
    raise ex
  File "/tmp/tmp.6P9sMbBYKP/rdma-core/tests/test_mlx5_mkey.py", line 57, in create_mkey
    self.mkey = Mlx5Mkey(self.pd, self.mkey_create_flags, 3)
  File "mlx5dv_mkey.pyx", line 186, in pyverbs.providers.mlx5.mlx5dv_mkey.Mlx5Mkey.__init__
pyverbs.pyverbs_error.PyverbsRDMAError: Failed to create mkey. Errno: 93, Protocol not supported

----------------------------------------------------------------------
Ran 193 tests in 9.649s

FAILED (errors=6, skipped=71)
---
- TEST RESULT FOR rdma-core
-   Test:   Run pyverbs tests
-   Result: FAIL
-   Return: 1
---
/mnt/tests/kernel/infiniband/pyverbs-tests
---



Expected results:

This is per another MLX5 IB/ROCE results in rdma-virt-02/03

test_tx_packet_reformat (tests.test_mlx5_flow.Mlx5MatcherTest) ... skipped 'NIC flow table does not support reformat'
test_raw_modify_lag_port (tests.test_mlx5_lag_affinity.LagPortTestCase) ... skipped 'Must be run by root on Ethernet link layer'
test_rc_modify_lag_port (tests.test_mlx5_lag_affinity.LagPortTestCase) ... skipped 'Set LAG affinity is not supported on this device'
test_ud_modify_lag_port (tests.test_mlx5_lag_affinity.LagPortTestCase) ... skipped 'Set LAG affinity is not supported on this device'
test_mkey_interleaved (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register an interleaved memory layout using this mkey and ... ok
test_mkey_interleaved_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, configure it with interleaved memory layout using the new ... ok
test_mkey_list (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register a memory layout using this mkey and then perform ... ok
test_mkey_list_bad_flow (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, register a memory layout using this mkey and then try to ... ok
test_mkey_list_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest)
Create Mkeys, configure it with memory layout using the new API and ... ok
test_mkey_sig_crc (tests.test_mlx5_mkey.Mlx5MkeyTest)


Additional info:

The hosts of rdma-dev-19/20 has the following info:

linux-firmware-20210315-120.el9.noarch
==> /sys/class/infiniband/mlx5_2/fw_ver <==
12.23.1020

==> /sys/class/infiniband/mlx5_3/fw_ver <==
12.23.1020

==> /sys/class/infiniband/mlx5_bond_0/fw_ver <==
14.23.1020
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
04:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]

o rdma-qe-06/07

CA 'mlx5_0'
	CA type: MT4113
	Number of ports: 2
	Firmware version: 10.16.1200
	Hardware version: 0
	Node GUID: 0xf452140300085d60
	System image GUID: 0xf452140300085d60
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 16
		LMC: 0
		SM lid: 3
		Capability mask: 0x26596848
		Port GUID: 0xf452140300085d60
		Link layer: InfiniBand
	Port 2:
		State: Active
		Physical state: LinkUp
		Rate: 56
		Base lid: 12
		LMC: 0
		SM lid: 1
		Capability mask: 0x26596848
		Port GUID: 0xf452140300085d68
		Link layer: InfiniBand

==============================================

o the MLX5 devices with SUCCESSFUL pyverbs-tests

---------------
rdma-virt-02/03
---------------

==> /sys/class/infiniband/mlx5_0/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_1/fw_ver <==
12.28.2006

==> /sys/class/infiniband/mlx5_bond_0/fw_ver <==
14.29.2002
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
04:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
06:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
06:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

---------------
rdma-dev-21/22
---------------

rdma-core-34.0-4.el9.x86_64
linux-firmware-20210315-120.el9.noarch
==> /sys/class/infiniband/mlx5_0/fw_ver <==
12.28.1002

==> /sys/class/infiniband/mlx5_1/fw_ver <==
12.28.1002

==> /sys/class/infiniband/mlx5_2/fw_ver <==
12.28.1002
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
82:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
Architecture:                    x86_64
Comment 3 RHEL Program Management 2022-12-14 07:27:54 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.