Bug 2211464

Summary:	Perftest versions prior to 23.04 missing Intel Parameters
Product:	Red Hat Enterprise Linux 8	Reporter:	ivan.d.barrera
Component:	perftest	Assignee:	Kamal Heib <kheib>
Status:	VERIFIED ---	QA Contact:	Brian Chae <bchae>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	8.8	CC:	dledford, hwkernel-mgr, ivan.d.barrera, kheib, rdma-dev-team, shiraz.saleem, tmichael
Target Milestone:	rc	Keywords:	Triaged
Target Release:	8.9
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	perftest-23.04.0.0.23-2.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2224042 (view as bug list)		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description ivan.d.barrera 2023-05-31 17:19:40 UTC

Description of problem:
Perftest versions prior to April 2023 are missing Intel parameters to run properly with default flags.

Version-Release number of selected component (if applicable):
https://github.com/linux-rdma/perftest/releases
Perftest version 4.5-0.20 and older missing parameters.

How reproducible:
Run perftest on Intel devices with default flags.

Steps to Reproduce:
on a "server" run: ib_send_lat
on a "client" run ib_send_lat IP_address_of_client

Actual results:
Both sides report:
Unable to create QP.
Failed to create QP.
 Couldn't create IB resources

Expected results:
Test should run, report latency

Additional info:
Default values are based on capabilities from a different device.
This has been corrected on the latest perftest release: perftest-23.04.0-0.23
https://github.com/linux-rdma/perftest/releases/tag/23.04.0-0.23

The request here is to please update the distro perftest versions to 23.04.0-0.23 or newer moving forward.

Thank you.

Comment 1 Brian Chae 2023-07-30 15:59:18 UTC

When perftest-23.04.0.0.23-2.el8 is tested on E810 iRDMA IW device, it still could not pass with defaults.

[root@rdma-dev-30 ~]$ rpm -q perftest
perftest-23.04.0.0.23-2.el8.x86_64
[root@rdma-dev-30 ~]$ 
                    
 
[root@rdma-dev-31 ~]$ rpm -q perftest
perftest-23.04.0.0.23-2.el8.x86_64
[root@rdma-dev-31 ~]$
 
 
[root@rdma-dev-30 ~]$ ib_send_lat
 Port number 1 state is Down
 Couldn't set the link layer
 Couldn't get context for the device

[root@rdma-dev-31 ~]$ ib_send_lat 172.31.50.130
Couldn't connect to 172.31.50.130:18515
Unable to open file descriptor for socket connection Unable to init the socket connection
[root@rdma-dev-31 ~]$

======================================

However, on the other HCAs, like MLX5 IB, perftest ran with defaults fine.

Upgraded:
  perftest-23.04.0.0.23-2.el8.x86_64


[root@rdma-perf-02 ~]$ rpm -q perftest
perftest-23.04.0.0.23-2.el8.x86_64
[root@rdma-perf-02 ~]$ 
 
[root@rdma-perf-03 ~]$ rpm -q perftest
perftest-23.04.0.0.23-2.el8.x86_64
[root@rdma-perf-03 ~]$ 

[root@rdma-perf-02 ~]$ ib_send_lat

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 RX depth        : 512
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 236[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x06 QPN 0x00d3 PSN 0xa9af28
 remote address: LID 0x05 QPN 0x00d3 PSN 0x9f0b3a
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
Conflicting CPU frequency values detected: 3000.000000 != 2693.424000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 3000.000000 != 2693.463000. CPU Frequency is not max.
 2       1000          1.21           4.23         1.53                1.46             0.07            2.01                    4.23
---------------------------------------------------------------------------------------
[root@rdma-perf-02 ~]$


[root@rdma-perf-03 ~]$ ib_send_lat 172.31.0.182
---------------------------------------------------------------------------------------
                    Send Latency Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 236[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x05 QPN 0x00d3 PSN 0x9f0b3a
 remote address: LID 0x06 QPN 0x00d3 PSN 0xa9af28
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec]
Conflicting CPU frequency values detected: 3000.000000 != 1243.832000. CPU Frequency is not max.
Conflicting CPU frequency values detected: 3000.000000 != 2194.703000. CPU Frequency is not max.
 2       1000          1.10           4.28         1.50                1.46             0.07            1.79                    4.28
---------------------------------------------------------------------------------------
[root@rdma-perf-03 ~]$

====================================


So, the perftest commands with just defaults still did not work.
Setting this bug back to Assigned state.

Comment 2 Brian Chae 2023-07-31 15:39:45 UTC

After discussion with Kamal in DEV, the first device, irdma0, was in DOWN state, the default for "ib_send_lat" command would select it and would have the errors. Hence, with rdma-dev-30/31 pair, we cannot test for default flags. We will just go with RDMA sanity and perftest tests results.

1. perftest package : perftest-23.04.0.0.23-2.el8.x86_64
2. tested HW :
   MLX5 IB0, IB1, RoCE
   E810 iRDMA
3. RDMA test suites tested : sanity, perftest
4. Results

1. E810 iRDMA iWARP

sanity test results on rdma-dev-30/rdma-dev-31 & Beaker job J:8131229:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, i40e, iw, E810-C & irdma1
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ibstatus reported expected HCA rate
      PASS |      0 | vlan i810_iw.81 create/delete
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      SKIP |    777 | ibsrpdm
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.50.131
      PASS |      0 | ping6 self - fe80::b696:91ff:fead:8aa9%i810_iw
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.50.130
      PASS |      0 | ping6 server - fe80::b696:91ff:fead:8589%i810_iw
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong        [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv       [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-EXT Window           [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-osu_get_bw
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      FAIL |    255 | rcopy                                    [ known issue -  bz 2117150 ]
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw
      PASS |      0 | ib_write_bw
      PASS |      0 | iser login
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      FAIL |    124 | iser write 1G                            [ known issue - on-off issue ]
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK


perftest test results on rdma-dev-30/rdma-dev-31 & Beaker job J:8131229:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, i40e, iw, E810-C & irdma1
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_lat RC
Checking for failures and known issues:
  no test failures

2. MLX5 IB0, IB1, RoCE

sanity test results on rdma-dev-19/rdma-dev-20 & Beaker job J:8131247:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib0, ConnectX-4 & mlx5_2
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | load module mlx5_ib
      PASS |      0 | load module mlx5_core
      PASS |      0 | enable opensm
      PASS |      0 | restart opensm
      PASS |      0 | osmtest -f c -g 0x248a07030049d468
      PASS |      0 | stop opensm
      PASS |      0 | disable opensm
      FAIL |      1 | ibstatus reported expected HCA rate
      PASS |      0 | pkey mlx5_ib0.8080 create/delete
      PASS |      0 | /usr/sbin/ibstat
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      PASS |      0 | /usr/sbin/ibsrpdm -vc
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.0.120
      PASS |      0 | ping6 self - fe80::268a:703:49:d468%mlx5_ib0
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.0.119
      PASS |      0 | ping6 server - fe80::268a:703:49:d338%mlx5_ib0
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv
      FAIL |      1 | openmpi mpitests-IMB-EXT Window
      FAIL |      1 | openmpi mpitests-osu_get_bw
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      PASS |      0 | rcopy
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw
      PASS |      0 | ib_write_bw
      PASS |      0 | iser login
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      PASS |      0 | iser write 1G
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK


sanity test results on rdma-dev-19/rdma-dev-20 & Beaker job J:8131247:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib1, ConnectX-4 & mlx5_3
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | load module mlx5_ib
      PASS |      0 | load module mlx5_core
      PASS |      0 | enable opensm
      PASS |      0 | restart opensm
      PASS |      0 | osmtest -f c -g 0x248a07030049d469
      PASS |      0 | stop opensm
      PASS |      0 | disable opensm
      PASS |      0 | ibstatus reported expected HCA rate
      PASS |      0 | pkey mlx5_ib1.8080 create/delete
      PASS |      0 | /usr/sbin/ibstat
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      PASS |      0 | /usr/sbin/ibsrpdm -vc
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.1.120
      PASS |      0 | ping6 self - fe80::268a:703:49:d469%mlx5_ib1
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.1.119
      PASS |      0 | ping6 server - fe80::268a:703:49:d339%mlx5_ib1
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv
      FAIL |      1 | openmpi mpitests-IMB-EXT Window
      FAIL |      1 | openmpi mpitests-osu_get_bw
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      PASS |      0 | rcopy
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw
      PASS |      0 | ib_write_bw
      PASS |      0 | iser login
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      PASS |      0 | iser write 1G
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK


sanity test results on rdma-dev-19/rdma-dev-20 & Beaker job J:8131247:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, roce.45, ConnectX-4 Lx & mlx5_bond_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | load module mlx5_ib
      PASS |      0 | load module mlx5_core
      FAIL |      1 | ibstatus reported expected HCA rate
      PASS |      0 | /usr/sbin/ibstat
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      SKIP |    777 | ibsrpdm
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.45.120
      PASS |      0 | ping6 self - fe80::7efe:90ff:fecb:762a%mlx5_team_ro.45
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.45.119
      PASS |      0 | ping6 server - fe80::7efe:90ff:fecb:743a%mlx5_team_ro.45
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv
      FAIL |      1 | openmpi mpitests-IMB-EXT Window
      FAIL |      1 | openmpi mpitests-osu_get_bw
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      PASS |      0 | rcopy
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw
      PASS |      0 | ib_write_bw
      PASS |      0 | iser login
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      PASS |      0 | iser write 1G
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK


perftest test results on rdma-dev-19/rdma-dev-20 & Beaker job J:8131247:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib0, ConnectX-4 & mlx5_2
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_atomic_bw RC
      PASS |      0 | ib_atomic_bw XRC
      PASS |      0 | ib_atomic_lat RC
      PASS |      0 | ib_atomic_lat XRC
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_bw XRC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_read_lat XRC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_bw XRC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_send_lat XRC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_bw XRC
      PASS |      0 | ib_write_lat RC
      PASS |      0 | ib_write_lat XRC
Checking for failures and known issues:
  no test failures

perftest test results on rdma-dev-19/rdma-dev-20 & Beaker job J:8131247:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib1, ConnectX-4 & mlx5_3
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_atomic_bw RC
      PASS |      0 | ib_atomic_bw XRC
      PASS |      0 | ib_atomic_lat RC
      PASS |      0 | ib_atomic_lat XRC
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_bw XRC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_read_lat XRC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_bw XRC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_send_lat XRC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_bw XRC
      PASS |      0 | ib_write_lat RC
      PASS |      0 | ib_write_lat XRC
Checking for failures and known issues:
  no test failures

perftest test results on rdma-dev-19/rdma-dev-20 & Beaker job J:8131247:
4.18.0-506.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, roce.45, ConnectX-4 Lx & mlx5_bond_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_atomic_bw RC
      PASS |      0 | ib_atomic_lat RC
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_lat RC
      PASS |      0 | raw_ethernet_bw RC
      PASS |      0 | raw_ethernet_lat RC
Checking for failures and known issues:
  no test failures


o No new issues found on sanity and perftest results.
o Setting this bug as "verified:tested"

Comment 5 Brian Chae 2023-08-04 20:00:14 UTC

The verification for ON_QA was done as the following:

1. build tested : RHEL-8.9.0-20230804.23
2. perftest package tested: perftest-23.04.0.0.23-2.el8.x86_64.rpm 

+ [23-08-04 13:00:55] rpm -q perftest
perftest-23.04.0.0.23-2.el8.x86_64

3. HW tested : MLX5 IB0, MLX5 IB1, MLX5 RoCE, QEDR IW, QEDR RoCE, CXGB4 IW, BNXT RoCE, HFI OPA0
4. RDMA test suites tested : sanity, perftest
5. Results: Only MLX5 IB0, IB1, RoCE test results are attached. Other HW test results will not be shown in this comment section but have been verified as successful.

   a. sanity

sanity test results on rdma-dev-21/rdma-dev-22 & Beaker job J:8153957:
4.18.0-507.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib0, ConnectX-4 & mlx5_1
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | load module mlx5_ib
      PASS |      0 | load module mlx5_core
      PASS |      0 | enable opensm
      PASS |      0 | restart opensm
      PASS |      0 | osmtest -f c -g 0x248a07030049d75c
      PASS |      0 | stop opensm
      PASS |      0 | disable opensm
      FAIL |      1 | ibstatus reported expected HCA rate
      PASS |      0 | pkey mlx5_ib0.8080 create/delete
      PASS |      0 | /usr/sbin/ibstat
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      PASS |      0 | /usr/sbin/ibsrpdm -vc
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.0.122
      PASS |      0 | ping6 self - fe80::268a:703:49:d75c%mlx5_ib0
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.0.121
      PASS |      0 | ping6 server - fe80::268a:703:49:d4f0%mlx5_ib0
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong         [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv        [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-EXT Window            [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-osu_get_bw                [ known issue - bz 2221806 ]
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      PASS |      0 | rcopy
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw
      PASS |      0 | ib_write_bw
      PASS |      0 | iser login
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      PASS |      0 | iser write 1G
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK

sanity test results on rdma-dev-21/rdma-dev-22 & Beaker job J:8153957:
4.18.0-507.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib1, ConnectX-4 & mlx5_2
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | load module mlx5_ib
      PASS |      0 | load module mlx5_core
      PASS |      0 | enable opensm
      PASS |      0 | restart opensm
      PASS |      0 | osmtest -f c -g 0x248a07030049d75d
      PASS |      0 | stop opensm
      PASS |      0 | disable opensm
      PASS |      0 | ibstatus reported expected HCA rate
      PASS |      0 | pkey mlx5_ib1.8080 create/delete
      PASS |      0 | /usr/sbin/ibstat
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      PASS |      0 | /usr/sbin/ibsrpdm -vc
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.1.122
      PASS |      0 | ping6 self - fe80::268a:703:49:d75d%mlx5_ib1
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.1.121
      PASS |      0 | ping6 server - fe80::268a:703:49:d4f1%mlx5_ib1
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong         [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv        [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-EXT Window            [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-osu_get_bw                [ known issue - bz 2221806 ]
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      PASS |      0 | rcopy
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw 
      PASS |      0 | ib_write_bw 
      PASS |      0 | iser login 
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      PASS |      0 | iser write 1G
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK

sanity test results on rdma-dev-21/rdma-dev-22 & Beaker job J:8153957:
4.18.0-507.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, roce.45, ConnectX-4 & mlx5_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | load module mlx5_ib
      PASS |      0 | load module mlx5_core
      PASS |      0 | ibstatus reported expected HCA rate
      PASS |      0 | /usr/sbin/ibstat
      PASS |      0 | /usr/sbin/ibstatus
      PASS |      0 | systemctl start srp_daemon.service
      SKIP |    777 | ibsrpdm
      PASS |      0 | systemctl stop srp_daemon
      PASS |      0 | ping self - 172.31.45.122
      PASS |      0 | ping6 self - fe80::268a:7ff:fe56:b834%mlx5_roce.45
      PASS |      0 | /usr/share/pmix/test/pmix_test
      PASS |      0 | ping server - 172.31.45.121
      PASS |      0 | ping6 server - fe80::268a:7ff:fe4b:f094%mlx5_roce.45
      FAIL |      1 | openmpi mpitests-IMB-MPI1 PingPong         [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-IO S_Read_indv        [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-IMB-EXT Window            [ known issue - bz 2221806 ]
      FAIL |      1 | openmpi mpitests-osu_get_bw                [ known issue - bz 2221806 ]
      PASS |      0 | ip multicast addr
      PASS |      0 | rping
      PASS |      0 | rcopy
      PASS |      0 | ib_read_bw
      PASS |      0 | ib_send_bw
      PASS |      0 | ib_write_bw
      PASS |      0 | iser login
      PASS |      0 | mount /dev/sdb /iser
      PASS |      0 | iser write 1K
      PASS |      0 | iser write 1M
      PASS |      0 | iser write 1G
      PASS |      0 | nfsordma mount - XFS_EXT
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - XFS_EXT
      PASS |      0 | nfsordma mount - RAMDISK
      PASS |      0 | nfsordma - wrote [5KB, 5MB, 5GB in 1KB, 1MB, 1GB bs]
      PASS |      0 | nfsordma umount - RAMDISK

    b. perftest

perftest test results on rdma-dev-21/rdma-dev-22 & Beaker job J:8153957:
4.18.0-507.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib0, ConnectX-4 & mlx5_1
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_atomic_bw RC
      PASS |      0 | ib_atomic_bw XRC
      PASS |      0 | ib_atomic_lat RC
      PASS |      0 | ib_atomic_lat XRC
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_bw XRC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_read_lat XRC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_bw XRC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_send_lat XRC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_bw XRC
      PASS |      0 | ib_write_lat RC
      PASS |      0 | ib_write_lat XRC

perftest test results on rdma-dev-21/rdma-dev-22 & Beaker job J:8153957:
4.18.0-507.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, ib1, ConnectX-4 & mlx5_2
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_atomic_bw RC
      PASS |      0 | ib_atomic_bw XRC
      PASS |      0 | ib_atomic_lat RC
      PASS |      0 | ib_atomic_lat XRC
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_bw XRC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_read_lat XRC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_bw XRC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_send_lat XRC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_bw XRC
      PASS |      0 | ib_write_lat RC
      PASS |      0 | ib_write_lat XRC

perftest test results on rdma-dev-21/rdma-dev-22 & Beaker job J:8153957:
4.18.0-507.el8.x86_64, rdma-core-46.0-1.el8.1, mlx5, roce.45, ConnectX-4 & mlx5_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | ib_atomic_bw RC
      PASS |      0 | ib_atomic_lat RC
      PASS |      0 | ib_read_bw RC
      PASS |      0 | ib_read_lat RC
      PASS |      0 | ib_send_bw RC
      PASS |      0 | ib_send_lat RC
      PASS |      0 | ib_write_bw RC
      PASS |      0 | ib_write_lat RC
      PASS |      0 | raw_ethernet_bw RC
      PASS |      0 | raw_ethernet_lat RC


o No new issues were observed
o All sanity and pertest results were verified as successful

Setting this bug as Verified