Hide Forgot
Description of problem: Upstream released MVAPICH2 2.3.6 on 05/11/2021. Let's update it for RHEL-8.5. Version-Release number of selected component (if applicable): 2.3.6 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: MVAPICH2 2.3.6 (05/11/2021) * Features and Enhancements (since 2.3.5): - Support collective offload using Mellanox's SHARP for Reduce and Bcast - Enhanced tuning framework for Reduce and Bcast using SHARP - Enhanced performance for UD-Hybrid code - Add multi-rail support for UD-Hybrid code - Enhanced performance for shared-memory collectives - Enhanced job-startup performance for flux job launcher - Add support in mpirun_rsh to use srun daemons to launch jobs - Add support in mpirun_rsh to specify processes per node using '-ppn' option - Use PMI2 by default when SLURM is selected as process manager - Add support to use aligned memory allocations for multi-threaded applications - Thanks to Evan J. Danish @OSC for the report - Architecture detection and enhanced point-to-point tuning for Oracle BM.HPC2 cloud shape - Enhanced collective tuning for Frontera@TACC and Expanse@SDSC - Add support for GCC compiler v11 - Add support for Intel IFX compiler - Update hwloc v1 code to v1.11.14 - Update hwloc v2 code to v2.4.2 * Bug Fixes (since 2.3.5): - Updates to IME support in MVAPICH2 - Thanks to Bernd Schubert and Jean-Yves Vet @DDN for the patch - Improve error reporting in dlopen code path - Thanks to Matthew W. Anderson @INL for the report - Fix memory leak in collectives code path - Thanks to Matthew W. Anderson @INL and the PETSc team for the report and patch - Fix issues in DPM code - Thanks to Lana Deere @D2S Inc for the report - Fix issues when using sys_siglist array - Thanks to Jorge D'Elia @Universidad Nacional Del Litoral in Santa Fe, Argentina for the report - Fix issues with GCC v11 - Thanks to Honggang Li @RedHat for the report - Fix issues in Win_shared_alloc - Thanks to Adam Moody @LLNL for the report - Fix issues with HDF5 in ROMIO code - Thanks to Mark Dixon @Durham University for the report - Fix issues with srun based launch when SLURM hostfile is specified manually - Thanks to Greg Lee @LLNL for the report - Fix an issue with external32 datatypes being converted incorrectly - Thanks to Adam Moody @LLNL for the report - Fix issues in UD-Hybrid code path - Fix issues in MPI_Win_test leading to hangs in multi-rail scenarios - Fix issues in job startup code leading to degraded startup performance - Update code to work with any number of HCAs in a graceful fashion - Fix hang in shared memory code with stencil applications - Fix segmentation fault in finalize - Fix compilation warnings, memory leaks, and spelling mistakes
The verification has been done as part of RHEL8.5 CTC#1 test cycle for RDMA. 1. build and packages DISTRO=RHEL-8.5.0-20210609.n.3 + [21-06-14 06:53:38] cat /etc/redhat-release Red Hat Enterprise Linux release 8.5 Beta (Ootpa) + [21-06-14 06:53:38] uname -a Linux rdma-dev-20.lab.bos.redhat.com 4.18.0-310.el8.x86_64 #1 SMP Thu May 27 14:56:02 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux + [21-06-14 06:53:38] cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-310.el8.x86_64 root=UUID=6754d3e0-4dac-4c9a-9f19-2c0d10402740 ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=auto resume=UUID=d2356100-f7ee-485c-9db4-ddd3ac4bfc43 console=ttyS1,115200n81 + [21-06-14 06:53:38] rpm -q rdma-core linux-firmware rdma-core-35.0-1.el8.x86_64 linux-firmware-20201218-102.git05789708.el8.noarch Installed: mpitests-mvapich2-5.7-2.el8.x86_64 mvapich2-2.3.6-1.el8.x86_64 2. tested HCAs MLX4 IB, MLX5 IB 3. Result Though some MLX5 IB have failed on many benchmarks, other MLX4 and MLX5 IBs has tested successfully, as the following: Test results for mpi/mvapich2 on rdma-dev-20: 4.18.0-310.el8.x86_64, rdma-core-35.0-1.el8, mlx5, ib0, & mlx5_2 Result | Status | Test ---------+--------+------------------------------------ PASS | 0 | mvapich2 IMB-MPI1 PingPong mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 PingPing mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Sendrecv mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Exchange mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Bcast mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Allgather mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Allgatherv mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Gather mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Gatherv mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Scatter mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Scatterv mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Alltoall mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Alltoallv mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Reduce mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Reduce_scatter mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Allreduce mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 Barrier mpirun one_core PASS | 0 | mvapich2 IMB-IO S_Write_indv mpirun one_core PASS | 0 | mvapich2 IMB-IO S_Read_indv mpirun one_core PASS | 0 | mvapich2 IMB-IO S_Write_expl mpirun one_core PASS | 0 | mvapich2 IMB-IO S_Read_expl mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Write_indv mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Read_indv mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Write_expl mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Read_expl mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Write_shared mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Read_shared mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Write_priv mpirun one_core PASS | 0 | mvapich2 IMB-IO P_Read_priv mpirun one_core PASS | 0 | mvapich2 IMB-IO C_Write_indv mpirun one_core PASS | 0 | mvapich2 IMB-IO C_Read_indv mpirun one_core PASS | 0 | mvapich2 IMB-IO C_Write_expl mpirun one_core PASS | 0 | mvapich2 IMB-IO C_Read_expl mpirun one_core PASS | 0 | mvapich2 IMB-IO C_Write_shared mpirun one_core PASS | 0 | mvapich2 IMB-IO C_Read_shared mpirun one_core PASS | 0 | mvapich2 IMB-EXT Window mpirun one_core PASS | 0 | mvapich2 IMB-EXT Unidir_Put mpirun one_core PASS | 0 | mvapich2 IMB-EXT Unidir_Get mpirun one_core PASS | 0 | mvapich2 IMB-EXT Bidir_Get mpirun one_core PASS | 0 | mvapich2 IMB-EXT Bidir_Put mpirun one_core PASS | 0 | mvapich2 IMB-EXT Accumulate mpirun one_core PASS | 0 | mvapich2 IMB-NBC Ibcast mpirun one_core PASS | 0 | mvapich2 IMB-NBC Iallgather mpirun one_core PASS | 0 | mvapich2 IMB-NBC Iallgatherv mpirun one_core PASS | 0 | mvapich2 IMB-NBC Igather mpirun one_core PASS | 0 | mvapich2 IMB-NBC Igatherv mpirun one_core PASS | 0 | mvapich2 IMB-NBC Iscatter mpirun one_core PASS | 0 | mvapich2 IMB-NBC Iscatterv mpirun one_core PASS | 0 | mvapich2 IMB-NBC Ialltoall mpirun one_core PASS | 0 | mvapich2 IMB-NBC Ialltoallv mpirun one_core PASS | 0 | mvapich2 IMB-NBC Ireduce mpirun one_core PASS | 0 | mvapich2 IMB-NBC Ireduce_scatter mpirun one_core PASS | 0 | mvapich2 IMB-NBC Iallreduce mpirun one_core PASS | 0 | mvapich2 IMB-NBC Ibarrier mpirun one_core PASS | 0 | mvapich2 IMB-RMA Unidir_put mpirun one_core PASS | 0 | mvapich2 IMB-RMA Unidir_get mpirun one_core PASS | 0 | mvapich2 IMB-RMA Bidir_put mpirun one_core PASS | 0 | mvapich2 IMB-RMA Bidir_get mpirun one_core PASS | 0 | mvapich2 IMB-RMA One_put_all mpirun one_core PASS | 0 | mvapich2 IMB-RMA One_get_all mpirun one_core PASS | 0 | mvapich2 IMB-RMA All_put_all mpirun one_core PASS | 0 | mvapich2 IMB-RMA All_get_all mpirun one_core PASS | 0 | mvapich2 IMB-RMA Put_local mpirun one_core PASS | 0 | mvapich2 IMB-RMA Put_all_local mpirun one_core PASS | 0 | mvapich2 IMB-RMA Exchange_put mpirun one_core PASS | 0 | mvapich2 IMB-RMA Exchange_get mpirun one_core PASS | 0 | mvapich2 IMB-RMA Accumulate mpirun one_core PASS | 0 | mvapich2 IMB-RMA Get_accumulate mpirun one_core PASS | 0 | mvapich2 IMB-RMA Fetch_and_op mpirun one_core PASS | 0 | mvapich2 IMB-RMA Compare_and_swap mpirun one_core PASS | 0 | mvapich2 IMB-RMA Get_local mpirun one_core PASS | 0 | mvapich2 IMB-RMA Get_all_local mpirun one_core PASS | 0 | mvapich2 IMB-MPI1 PingPong mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 PingPing mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Sendrecv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Exchange mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Bcast mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Allgather mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Allgatherv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Gather mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Gatherv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Scatter mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Scatterv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Alltoall mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Alltoallv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Reduce mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Reduce_scatter mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Allreduce mpirun_rsh one_core PASS | 0 | mvapich2 IMB-MPI1 Barrier mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO S_Write_indv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO S_Read_indv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO S_Write_expl mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO S_Read_expl mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Write_indv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Read_indv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Write_expl mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Read_expl mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Write_shared mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Read_shared mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Write_priv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO P_Read_priv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO C_Write_indv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO C_Read_indv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO C_Write_expl mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO C_Read_expl mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO C_Write_shared mpirun_rsh one_core PASS | 0 | mvapich2 IMB-IO C_Read_shared mpirun_rsh one_core PASS | 0 | mvapich2 IMB-EXT Window mpirun_rsh one_core PASS | 0 | mvapich2 IMB-EXT Unidir_Put mpirun_rsh one_core PASS | 0 | mvapich2 IMB-EXT Unidir_Get mpirun_rsh one_core PASS | 0 | mvapich2 IMB-EXT Bidir_Get mpirun_rsh one_core PASS | 0 | mvapich2 IMB-EXT Bidir_Put mpirun_rsh one_core PASS | 0 | mvapich2 IMB-EXT Accumulate mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Ibcast mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Iallgather mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Iallgatherv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Igather mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Igatherv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Iscatter mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Iscatterv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Ialltoall mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Ialltoallv mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Ireduce mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Ireduce_scatter mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Iallreduce mpirun_rsh one_core PASS | 0 | mvapich2 IMB-NBC Ibarrier mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Unidir_put mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Unidir_get mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Bidir_put mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Bidir_get mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA One_put_all mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA One_get_all mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA All_put_all mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA All_get_all mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Put_local mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Put_all_local mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Exchange_put mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Exchange_get mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Accumulate mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Get_accumulate mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Fetch_and_op mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Compare_and_swap mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Get_local mpirun_rsh one_core PASS | 0 | mvapich2 IMB-RMA Get_all_local mpirun_rsh one_core PASS | 0 | mvapich2 OSU acc_latency mpirun one_core PASS | 0 | mvapich2 OSU allgather mpirun one_core PASS | 0 | mvapich2 OSU allgatherv mpirun one_core PASS | 0 | mvapich2 OSU allreduce mpirun one_core PASS | 0 | mvapich2 OSU alltoall mpirun one_core PASS | 0 | mvapich2 OSU alltoallv mpirun one_core PASS | 0 | mvapich2 OSU barrier mpirun one_core PASS | 0 | mvapich2 OSU bcast mpirun one_core PASS | 0 | mvapich2 OSU bibw mpirun one_core PASS | 0 | mvapich2 OSU bw mpirun one_core PASS | 0 | mvapich2 OSU cas_latency mpirun one_core PASS | 0 | mvapich2 OSU fop_latency mpirun one_core PASS | 0 | mvapich2 OSU gather mpirun one_core PASS | 0 | mvapich2 OSU gatherv mpirun one_core PASS | 0 | mvapich2 OSU get_acc_latency mpirun one_core PASS | 0 | mvapich2 OSU get_bw mpirun one_core PASS | 0 | mvapich2 OSU get_latency mpirun one_core PASS | 0 | mvapich2 OSU hello mpirun one_core PASS | 0 | mvapich2 OSU iallgather mpirun one_core PASS | 0 | mvapich2 OSU iallgatherv mpirun one_core PASS | 0 | mvapich2 OSU iallreduce mpirun one_core PASS | 0 | mvapich2 OSU ialltoall mpirun one_core PASS | 0 | mvapich2 OSU ialltoallv mpirun one_core PASS | 0 | mvapich2 OSU ialltoallw mpirun one_core PASS | 0 | mvapich2 OSU ibarrier mpirun one_core PASS | 0 | mvapich2 OSU ibcast mpirun one_core PASS | 0 | mvapich2 OSU igather mpirun one_core PASS | 0 | mvapich2 OSU igatherv mpirun one_core PASS | 0 | mvapich2 OSU init mpirun one_core PASS | 0 | mvapich2 OSU ireduce mpirun one_core PASS | 0 | mvapich2 OSU iscatter mpirun one_core PASS | 0 | mvapich2 OSU iscatterv mpirun one_core PASS | 0 | mvapich2 OSU latency mpirun one_core PASS | 0 | mvapich2 OSU latency_mp mpirun one_core PASS | 0 | mvapich2 OSU mbw_mr mpirun one_core PASS | 0 | mvapich2 OSU multi_lat mpirun one_core PASS | 0 | mvapich2 OSU put_bibw mpirun one_core PASS | 0 | mvapich2 OSU put_bw mpirun one_core PASS | 0 | mvapich2 OSU put_latency mpirun one_core PASS | 0 | mvapich2 OSU reduce mpirun one_core PASS | 0 | mvapich2 OSU reduce_scatter mpirun one_core PASS | 0 | mvapich2 OSU scatter mpirun one_core PASS | 0 | mvapich2 OSU scatterv mpirun one_core PASS | 0 | mvapich2 OSU acc_latency mpirun_rsh one_core PASS | 0 | mvapich2 OSU allgather mpirun_rsh one_core PASS | 0 | mvapich2 OSU allgatherv mpirun_rsh one_core PASS | 0 | mvapich2 OSU allreduce mpirun_rsh one_core PASS | 0 | mvapich2 OSU alltoall mpirun_rsh one_core PASS | 0 | mvapich2 OSU alltoallv mpirun_rsh one_core PASS | 0 | mvapich2 OSU barrier mpirun_rsh one_core PASS | 0 | mvapich2 OSU bcast mpirun_rsh one_core PASS | 0 | mvapich2 OSU bibw mpirun_rsh one_core PASS | 0 | mvapich2 OSU bw mpirun_rsh one_core PASS | 0 | mvapich2 OSU cas_latency mpirun_rsh one_core PASS | 0 | mvapich2 OSU fop_latency mpirun_rsh one_core PASS | 0 | mvapich2 OSU gather mpirun_rsh one_core PASS | 0 | mvapich2 OSU gatherv mpirun_rsh one_core PASS | 0 | mvapich2 OSU get_acc_latency mpirun_rsh one_core PASS | 0 | mvapich2 OSU get_bw mpirun_rsh one_core PASS | 0 | mvapich2 OSU get_latency mpirun_rsh one_core PASS | 0 | mvapich2 OSU hello mpirun_rsh one_core PASS | 0 | mvapich2 OSU iallgather mpirun_rsh one_core PASS | 0 | mvapich2 OSU iallgatherv mpirun_rsh one_core PASS | 0 | mvapich2 OSU iallreduce mpirun_rsh one_core PASS | 0 | mvapich2 OSU ialltoall mpirun_rsh one_core PASS | 0 | mvapich2 OSU ialltoallv mpirun_rsh one_core PASS | 0 | mvapich2 OSU ialltoallw mpirun_rsh one_core PASS | 0 | mvapich2 OSU ibarrier mpirun_rsh one_core PASS | 0 | mvapich2 OSU ibcast mpirun_rsh one_core PASS | 0 | mvapich2 OSU igather mpirun_rsh one_core PASS | 0 | mvapich2 OSU igatherv mpirun_rsh one_core PASS | 0 | mvapich2 OSU init mpirun_rsh one_core PASS | 0 | mvapich2 OSU ireduce mpirun_rsh one_core PASS | 0 | mvapich2 OSU iscatter mpirun_rsh one_core PASS | 0 | mvapich2 OSU iscatterv mpirun_rsh one_core PASS | 0 | mvapich2 OSU latency mpirun_rsh one_core FAIL | 1 | mvapich2 OSU latency_mp mpirun_rsh one_core PASS | 0 | mvapich2 OSU mbw_mr mpirun_rsh one_core PASS | 0 | mvapich2 OSU multi_lat mpirun_rsh one_core PASS | 0 | mvapich2 OSU put_bibw mpirun_rsh one_core PASS | 0 | mvapich2 OSU put_bw mpirun_rsh one_core PASS | 0 | mvapich2 OSU put_latency mpirun_rsh one_core PASS | 0 | mvapich2 OSU reduce mpirun_rsh one_core PASS | 0 | mvapich2 OSU reduce_scatter mpirun_rsh one_core PASS | 0 | mvapich2 OSU scatter mpirun_rsh one_core PASS | 0 | mvapich2 OSU scatterv mpirun_rsh one_core
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RDMA stack bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4412