Bug 1850084

Summary: mvapich2 needs to be rebuilt against hwloc-2.2.0
Product: Red Hat Enterprise Linux 8 Reporter: Prarit Bhargava <prarit>
Component: mvapich2Assignee: Honggang LI <honli>
Status: CLOSED ERRATA QA Contact: Afom T. Michael <tmichael>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.4CC: bchae, honli, jwboyer, linville, pkotvan, prarit, pvlasin, rdma-dev-team, tmichael
Target Milestone: rc   
Target Release: 8.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mvapich2-2.3.4-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 14:42:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1732988, 1841354    
Bug Blocks: 1850703, 1894575    

Description Prarit Bhargava 2020-06-23 14:05:00 UTC 
Description of problem: hwloc was updated to 2.2.0.  The mvapich2 specfile has

BuildRequires: python3-devel, perl-Digest-MD5, hwloc-devel, rdma-core-devel

Version-Release number of selected component (if applicable):


How reproducible: mvapich2-2.3.3-1.el8


Steps to Reproduce:
1. yum -y install mvapich2
2.
3.

Actual results:

[10:04 AM root@amd-daytona-03 ~]# yum -y install mvapich2
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Last metadata expiration check: 0:19:09 ago on Tue 23 Jun 2020 09:45:23 AM EDT.
Error: 
 Problem: cannot install the best candidate for the job
  - nothing provides libhwloc.so.5()(64bit) needed by mvapich2-2.3.3-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)



Expected results: mvapich2 installs.


Additional info: hwloc was upgraded to 2.2.0.
Comment 3 Honggang LI 2020-06-24 07:42:25 UTC 
mvapich2-2.3.3 was based on hwloc 1.x. I will update mvapich2-2.3.4 to address this bug.
Comment 13 Brian Chae 2020-10-22 18:19:32 UTC 
Tier2 MPI test suites tested MPICH benchmarks.

DISTRO=RHEL-8.4.0-20201016.d.0
Installed:
  mpich-3.3.2-9.el8.x86_64           mpitests-mpich-5.6.2-1.el8.x86_64          


Test resuilts:

https://beaker.engineering.redhat.com/jobs/4652665

Test results for mpi/mpich2 on rdma-dev-12:
4.18.0-240.5.el8.x86_64, rdma-core-29.0-3.el8, cxgb4, iw, & cxgb4_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mpich OSU acc_latency mpiexec one_core
      PASS |      0 | mpich OSU allgather mpiexec one_core
      PASS |      0 | mpich OSU allgatherv mpiexec one_core
      PASS |      0 | mpich OSU allreduce mpiexec one_core
      PASS |      0 | mpich OSU alltoall mpiexec one_core
      PASS |      0 | mpich OSU alltoallv mpiexec one_core
      PASS |      0 | mpich OSU barrier mpiexec one_core
      PASS |      0 | mpich OSU bcast mpiexec one_core
      PASS |      0 | mpich OSU bibw mpiexec one_core
      PASS |      0 | mpich OSU bw mpiexec one_core
      PASS |      0 | mpich OSU cas_latency mpiexec one_core
      PASS |      0 | mpich OSU fop_latency mpiexec one_core
      PASS |      0 | mpich OSU gather mpiexec one_core
      PASS |      0 | mpich OSU gatherv mpiexec one_core
      PASS |      0 | mpich OSU get_acc_latency mpiexec one_core
      PASS |      0 | mpich OSU get_bw mpiexec one_core
      PASS |      0 | mpich OSU get_latency mpiexec one_core
      PASS |      0 | mpich OSU hello mpiexec one_core
      PASS |      0 | mpich OSU iallgather mpiexec one_core
      PASS |      0 | mpich OSU iallgatherv mpiexec one_core
      PASS |      0 | mpich OSU iallreduce mpiexec one_core
      PASS |      0 | mpich OSU ialltoall mpiexec one_core
      PASS |      0 | mpich OSU ialltoallv mpiexec one_core
      PASS |      0 | mpich OSU ialltoallw mpiexec one_core
      PASS |      0 | mpich OSU ibarrier mpiexec one_core
      PASS |      0 | mpich OSU ibcast mpiexec one_core
      PASS |      0 | mpich OSU igather mpiexec one_core
      PASS |      0 | mpich OSU igatherv mpiexec one_core
      PASS |      0 | mpich OSU init mpiexec one_core
      PASS |      0 | mpich OSU ireduce mpiexec one_core
      PASS |      0 | mpich OSU iscatter mpiexec one_core
      PASS |      0 | mpich OSU iscatterv mpiexec one_core
      PASS |      0 | mpich OSU latency mpiexec one_core
      PASS |      0 | mpich OSU mbw_mr mpiexec one_core
      PASS |      0 | mpich OSU multi_lat mpiexec one_core
      PASS |      0 | mpich OSU put_bibw mpiexec one_core
      PASS |      0 | mpich OSU put_bw mpiexec one_core
      PASS |      0 | mpich OSU put_latency mpiexec one_core
      PASS |      0 | mpich OSU reduce mpiexec one_core
      PASS |      0 | mpich OSU reduce_scatter mpiexec one_core
      PASS |      0 | mpich OSU scatter mpiexec one_core
      PASS |      0 | mpich OSU scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPong mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPing mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Sendrecv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Exchange mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Bcast mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoall mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoallv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allreduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Barrier mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-EXT Window mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibcast mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatterv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoall mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoallv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallreduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibarrier mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_all_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Fetch_and_op mpiexec one_core
      PASS |      0 | mpich IMB-RMA Compare_and_swap mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_all_local mpiexec one_core

Checking for failures and known issues:
  no test failures
Comment 14 Brian Chae 2020-10-23 09:51:08 UTC 
The above was for OPENMPI run results. This is to provide MVAPICH2 run results.

The result was FAILED but this was with known issue from RHEL8.3 -bz 1878829; NO NEW issues found in RHEL8.4


DISTRO=RHEL-8.4.0-20201016.d.0
Installed:
  mpitests-mvapich2-5.6.2-1.el8.x86_64        mvapich2-2.3.4-1.el8.x86_64       

Test Results:

https://beaker.engineering.redhat.com/jobs/4652661

  - all benchmarks with "mpirun" PASSED
  - all benchmarks with "mpirun_rsh" FAILED with RC 1 (timedout)

Additional info:

No such issues, meaning all benchmarks with both "mpirun" and "mpirun_rsh" PASSED when both server & client hosts are with the following:

  - rdma-qe-06/rdma-qe-07
    mlx5 MT27600 CIB ib0/ib1

   - rdma-dev-10/11
     mlx4 MT27500 CX-3 ib0/ib1

   - rdma-dev-00/01
     mlx4 MT27500 CX-3 ib0/ib1

   - rdma-perf-00/01
     mlx4 MT27500 CX-3 ib0
Comment 17 Afom T. Michael 2020-12-24 21:55:01 UTC 
Tests have passed as shown below and hence moving to verified. Installation of mvapich2 depends on hwloc-libs.

[root@rdma-cki-05 SPECS]$ grep -E "Version|hwloc" mvapich2.spec 
Version: 2.3.5
Patch1:  0001-mvapich23-unbundle-contrib-hwloc.patch
BuildRequires: python3-devel, perl-Digest-MD5, hwloc-devel, rdma-core-devel
# bundled hwloc, knem kernel module
    --with-hwloc=v2 \
    --with-hwloc=v2
- Build against hwloc-2.2.0
[root@rdma-cki-05 SPECS]$

[root@rdma-cki-05 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
[root@rdma-cki-05 ~]$ uname -r
4.18.0-266.el8.x86_64
[root@rdma-cki-05 ~]$ rpm -qa | grep -E "rdma|ibverbs|openmpi|mvapich|hwloc|ucx"
hwloc-libs-2.2.0-1.el8.x86_64
libibverbs-32.0-3.el8.x86_64
mpitests-mvapich2-5.6.3-1.el8.x86_64
librdmacm-32.0-3.el8.x86_64
ucx-1.9.0-1.el8.x86_64
rdma-core-devel-32.0-3.el8.x86_64
librdmacm-utils-32.0-3.el8.x86_64
rdma-core-32.0-3.el8.x86_64
libibverbs-utils-32.0-3.el8.x86_64
mvapich2-2.3.5-1.el8.x86_64
[root@rdma-cki-05 ~]$ ibv_devinfo 
hca_id:	hfi1_0
	transport:			InfiniBand (0)
	fw_ver:				1.27.0
	node_guid:			0011:7501:0167:1ecf
	sys_image_guid:			0011:7501:0167:1ecf
	vendor_id:			0x1175
	vendor_part_id:			9456
	hw_ver:				0x10
	board_id:			Intel Omni-Path Host Fabric Interface Adapter 100 Series
	phys_port_cnt:			1
		port:	1
			state:			PORT_ACTIVE (4)
			max_mtu:		4096 (5)
			active_mtu:		4096 (5)
			sm_lid:			1
			port_lid:		10
			port_lmc:		0x00
			link_layer:		InfiniBand


[root@rdma-cki-05 ~]$ yum remove mvapich2
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Dependencies resolved.
===========================================================================================================================================
 Package                              Architecture              Version                         Repository                            Size
===========================================================================================================================================
Removing:
 mvapich2                             x86_64                    2.3.5-1.el8                     @brew                                 17 M
Removing dependent packages:
 mpitests-mvapich2                    x86_64                    5.6.3-1.el8                     @beaker-AppStream                    5.7 M
Removing unused dependencies:
 hwloc-libs                           x86_64                    2.2.0-1.el8                     @beaker-BaseOS                       2.9 M

Transaction Summary
===========================================================================================================================================
Remove  3 Packages

Freed space: 25 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                   1/1 
  Erasing          : mpitests-mvapich2-5.6.3-1.el8.x86_64                                                                              1/3 
  Erasing          : mvapich2-2.3.5-1.el8.x86_64                                                                                       2/3 
  Erasing          : hwloc-libs-2.2.0-1.el8.x86_64                                                                                     3/3 
  Running scriptlet: hwloc-libs-2.2.0-1.el8.x86_64                                                                                     3/3 
  Verifying        : hwloc-libs-2.2.0-1.el8.x86_64                                                                                     1/3 
  Verifying        : mpitests-mvapich2-5.6.3-1.el8.x86_64                                                                              2/3 
  Verifying        : mvapich2-2.3.5-1.el8.x86_64                                                                                       3/3 
Installed products updated.

Removed:
  hwloc-libs-2.2.0-1.el8.x86_64               mpitests-mvapich2-5.6.3-1.el8.x86_64               mvapich2-2.3.5-1.el8.x86_64              

Complete!
[root@rdma-cki-05 ~]$ yum install mvapich2
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Last metadata expiration check: 3:59:01 ago on Thu 24 Dec 2020 12:43:36 PM EST.
Dependencies resolved.
===========================================================================================================================================
 Package                          Architecture                 Version                           Repository                           Size
===========================================================================================================================================
Installing:
 mvapich2                         x86_64                       2.3.5-1.el8                       brew                                3.8 M
Installing dependencies:
 hwloc-libs                       x86_64                       2.2.0-1.el8                       beaker-BaseOS                       2.0 M

Transaction Summary
===========================================================================================================================================
Install  2 Packages

Total download size: 5.8 M
Installed size: 19 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): hwloc-libs-2.2.0-1.el8.x86_64.rpm                                                                    22 MB/s | 2.0 MB     00:00    
(2/2): mvapich2-2.3.5-1.el8.x86_64.rpm                                                                      33 MB/s | 3.8 MB     00:00    
-------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                       50 MB/s | 5.8 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                   1/1 
  Installing       : hwloc-libs-2.2.0-1.el8.x86_64                                                                                     1/2 
  Installing       : mvapich2-2.3.5-1.el8.x86_64                                                                                       2/2 
  Running scriptlet: mvapich2-2.3.5-1.el8.x86_64                                                                                       2/2 
  Verifying        : hwloc-libs-2.2.0-1.el8.x86_64                                                                                     1/2 
  Verifying        : mvapich2-2.3.5-1.el8.x86_64                                                                                       2/2 
Installed products updated.

Installed:
  hwloc-libs-2.2.0-1.el8.x86_64                                         mvapich2-2.3.5-1.el8.x86_64                                        

Complete!
[root@rdma-cki-05 ~]$ yum install mpitests-mvapich2
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

Last metadata expiration check: 3:59:32 ago on Thu 24 Dec 2020 12:43:36 PM EST.
Dependencies resolved.
===========================================================================================================================================
 Package                              Architecture              Version                          Repository                           Size
===========================================================================================================================================
Installing:
 mpitests-mvapich2                    x86_64                    5.6.3-1.el8                      beaker-AppStream                    890 k

Transaction Summary
===========================================================================================================================================
Install  1 Package

Total download size: 890 k
Installed size: 5.7 M
Is this ok [y/N]: y
Downloading Packages:
mpitests-mvapich2-5.6.3-1.el8.x86_64.rpm                                                                    20 MB/s | 890 kB     00:00    
-------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                       20 MB/s | 890 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                   1/1 
  Installing       : mpitests-mvapich2-5.6.3-1.el8.x86_64                                                                              1/1 
  Running scriptlet: mpitests-mvapich2-5.6.3-1.el8.x86_64                                                                              1/1 
  Verifying        : mpitests-mvapich2-5.6.3-1.el8.x86_64                                                                              1/1 
Installed products updated.

Installed:
  mpitests-mvapich2-5.6.3-1.el8.x86_64                                                                                                     

Complete!
[root@rdma-cki-05 ~]$


Test results for mpi/mvapich2 on rdma-cki-05:
4.18.0-266.el8.x86_64, rdma-core-32.0-3.el8, hfi1, opa0, & hfi1_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mvapich2 IMB-MPI1 PingPong mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 PingPing mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Sendrecv mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Exchange mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Bcast mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Allgather mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Allgatherv mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Gather mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Gatherv mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Scatter mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Scatterv mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Alltoall mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Alltoallv mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Reduce mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Reduce_scatter mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Allreduce mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 Barrier mpirun one_core
      PASS |      0 | mvapich2 IMB-IO S_Write_indv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO S_Read_indv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO S_Write_expl mpirun one_core
      PASS |      0 | mvapich2 IMB-IO S_Read_expl mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_indv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_indv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_expl mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_expl mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_shared mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_shared mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_priv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_priv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO C_Write_indv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO C_Read_indv mpirun one_core
      PASS |      0 | mvapich2 IMB-IO C_Write_expl mpirun one_core
      PASS |      0 | mvapich2 IMB-IO C_Read_expl mpirun one_core
      PASS |      0 | mvapich2 IMB-IO C_Write_shared mpirun one_core
      PASS |      0 | mvapich2 IMB-IO C_Read_shared mpirun one_core
      PASS |      0 | mvapich2 IMB-EXT Window mpirun one_core
      PASS |      0 | mvapich2 IMB-EXT Unidir_Put mpirun one_core
      PASS |      0 | mvapich2 IMB-EXT Unidir_Get mpirun one_core
      PASS |      0 | mvapich2 IMB-EXT Bidir_Get mpirun one_core
      PASS |      0 | mvapich2 IMB-EXT Bidir_Put mpirun one_core
      PASS |      0 | mvapich2 IMB-EXT Accumulate mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Ibcast mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Iallgather mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Iallgatherv mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Igather mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Igatherv mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Iscatter mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Iscatterv mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Ialltoall mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Ialltoallv mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Ireduce mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Ireduce_scatter mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Iallreduce mpirun one_core
      PASS |      0 | mvapich2 IMB-NBC Ibarrier mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Unidir_put mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Unidir_get mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Bidir_put mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Bidir_get mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA One_put_all mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA One_get_all mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA All_put_all mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA All_get_all mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Put_local mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Put_all_local mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Exchange_put mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Exchange_get mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Accumulate mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Get_accumulate mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Fetch_and_op mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Compare_and_swap mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Get_local mpirun one_core
      PASS |      0 | mvapich2 IMB-RMA Get_all_local mpirun one_core
      PASS |      0 | mvapich2 IMB-MPI1 PingPong mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 PingPing mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Sendrecv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Exchange mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Bcast mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Allgather mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Allgatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Gather mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Gatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Scatter mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Scatterv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Alltoall mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Alltoallv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Reduce mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Reduce_scatter mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Allreduce mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-MPI1 Barrier mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO S_Write_indv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO S_Read_indv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO S_Write_expl mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO S_Read_expl mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_indv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_indv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_expl mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_expl mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_shared mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_shared mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Write_priv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO P_Read_priv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO C_Write_indv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO C_Read_indv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO C_Write_expl mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO C_Read_expl mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO C_Write_shared mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-IO C_Read_shared mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-EXT Window mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-EXT Unidir_Put mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-EXT Unidir_Get mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-EXT Bidir_Get mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-EXT Bidir_Put mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-EXT Accumulate mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Ibcast mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Iallgather mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Iallgatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Igather mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Igatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Iscatter mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Iscatterv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Ialltoall mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Ialltoallv mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Ireduce mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Ireduce_scatter mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Iallreduce mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-NBC Ibarrier mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Unidir_put mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Unidir_get mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Bidir_put mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Bidir_get mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA One_put_all mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA One_get_all mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA All_put_all mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA All_get_all mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Put_local mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Put_all_local mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Exchange_put mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Exchange_get mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Accumulate mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Get_accumulate mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Fetch_and_op mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Compare_and_swap mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Get_local mpirun_rsh one_core
      PASS |      0 | mvapich2 IMB-RMA Get_all_local mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU acc_latency mpirun one_core
      PASS |      0 | mvapich2 OSU allgather mpirun one_core
      PASS |      0 | mvapich2 OSU allgatherv mpirun one_core
      PASS |      0 | mvapich2 OSU allreduce mpirun one_core
      PASS |      0 | mvapich2 OSU alltoall mpirun one_core
      PASS |      0 | mvapich2 OSU alltoallv mpirun one_core
      PASS |      0 | mvapich2 OSU barrier mpirun one_core
      PASS |      0 | mvapich2 OSU bcast mpirun one_core
      PASS |      0 | mvapich2 OSU bibw mpirun one_core
      PASS |      0 | mvapich2 OSU bw mpirun one_core
      PASS |      0 | mvapich2 OSU cas_latency mpirun one_core
      PASS |      0 | mvapich2 OSU fop_latency mpirun one_core
      PASS |      0 | mvapich2 OSU gather mpirun one_core
      PASS |      0 | mvapich2 OSU gatherv mpirun one_core
      PASS |      0 | mvapich2 OSU get_acc_latency mpirun one_core
      PASS |      0 | mvapich2 OSU get_bw mpirun one_core
      PASS |      0 | mvapich2 OSU get_latency mpirun one_core
      PASS |      0 | mvapich2 OSU hello mpirun one_core
      PASS |      0 | mvapich2 OSU iallgather mpirun one_core
      PASS |      0 | mvapich2 OSU iallgatherv mpirun one_core
      PASS |      0 | mvapich2 OSU iallreduce mpirun one_core
      PASS |      0 | mvapich2 OSU ialltoall mpirun one_core
      PASS |      0 | mvapich2 OSU ialltoallv mpirun one_core
      PASS |      0 | mvapich2 OSU ialltoallw mpirun one_core
      PASS |      0 | mvapich2 OSU ibarrier mpirun one_core
      PASS |      0 | mvapich2 OSU ibcast mpirun one_core
      PASS |      0 | mvapich2 OSU igather mpirun one_core
      PASS |      0 | mvapich2 OSU igatherv mpirun one_core
      PASS |      0 | mvapich2 OSU init mpirun one_core
      PASS |      0 | mvapich2 OSU ireduce mpirun one_core
      PASS |      0 | mvapich2 OSU iscatter mpirun one_core
      PASS |      0 | mvapich2 OSU iscatterv mpirun one_core
      PASS |      0 | mvapich2 OSU latency mpirun one_core
      PASS |      0 | mvapich2 OSU latency_mp mpirun one_core
      PASS |      0 | mvapich2 OSU mbw_mr mpirun one_core
      PASS |      0 | mvapich2 OSU multi_lat mpirun one_core
      PASS |      0 | mvapich2 OSU put_bibw mpirun one_core
      PASS |      0 | mvapich2 OSU put_bw mpirun one_core
      PASS |      0 | mvapich2 OSU put_latency mpirun one_core
      PASS |      0 | mvapich2 OSU reduce mpirun one_core
      PASS |      0 | mvapich2 OSU reduce_scatter mpirun one_core
      PASS |      0 | mvapich2 OSU scatter mpirun one_core
      PASS |      0 | mvapich2 OSU scatterv mpirun one_core
      PASS |      0 | mvapich2 OSU acc_latency mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU allgather mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU allgatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU allreduce mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU alltoall mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU alltoallv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU barrier mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU bcast mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU bibw mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU bw mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU cas_latency mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU fop_latency mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU gather mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU gatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU get_acc_latency mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU get_bw mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU get_latency mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU hello mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU iallgather mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU iallgatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU iallreduce mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU ialltoall mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU ialltoallv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU ialltoallw mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU ibarrier mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU ibcast mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU igather mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU igatherv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU init mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU ireduce mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU iscatter mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU iscatterv mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU latency mpirun_rsh one_core
      FAIL |      1 | mvapich2 OSU latency_mp mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU mbw_mr mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU multi_lat mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU put_bibw mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU put_bw mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU put_latency mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU reduce mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU reduce_scatter mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU scatter mpirun_rsh one_core
      PASS |      0 | mvapich2 OSU scatterv mpirun_rsh one_core
Comment 19 errata-xmlrpc 2021-05-18 14:42:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RDMA stack bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1594