Bug 1732982

Summary:	[RHEL-8.2] mpich: update to upstream 3.3.x series release
Product:	Red Hat Enterprise Linux 8	Reporter:	Jarod Wilson <jarod>
Component:	mpich	Assignee:	Honggang LI <honli>
Status:	CLOSED ERRATA	QA Contact:	Afom T. Michael <tmichael>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	8.2	CC:	bchae, jarod, jwboyer, linville, mschmidt, peter.georg, rdma-dev-team, zguo
Target Milestone:	rc
Target Release:	8.4
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	mpich-3.3.2-8.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-18 14:42:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1732988, 1841354
Bug Blocks:	1657316, 1850080, 1850703, 1894575

Description Jarod Wilson 2019-07-24 21:38:47 UTC

Upstream mpich 3.3.0 was released back in September of 2018, and a 3.3.1 stable update has since been released, and all mpich users are encouraged to upgrade. The new mpich version includes new hardware support for more recent hardware via OFI and UCX modules, which feature much lower CPU overhead with more functionality offloaded to the hardware.

Note: mpich 3.3.x requires hwloc v2.0.x or later as well, so a subsequent bug will be filed requesting that update as well.

Comment 13 Brian Chae 2020-10-22 17:38:00 UTC

Tier2 MPI test suite tested successfully on MPICH & OPENMPI benchmarks.

DISTRO=RHEL-8.4.0-20201016.d.0
Installed:
  mpich-3.3.2-9.el8.x86_64           mpitests-mpich-5.6.2-1.el8.x86_64

Test results:

https://beaker.engineering.redhat.com/jobs/4653094

Comment 14 Brian Chae 2020-10-22 18:03:08 UTC

(In reply to Brian Chae from comment #13)
> Tier2 MPI test suite tested successfully on MPICH & OPENMPI benchmarks.
> 
> DISTRO=RHEL-8.4.0-20201016.d.0
> Installed:
>   mpich-3.3.2-9.el8.x86_64           mpitests-mpich-5.6.2-1.el8.x86_64
> 
> Test results:
> 
> https://beaker.engineering.redhat.com/jobs/4653094

Test results for mpi/mpich2 on rdma-virt-03:
4.18.0-240.5.el8.x86_64, rdma-core-29.0-3.el8, mlx5, ib1, & mlx5_1
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mpich OSU acc_latency mpiexec one_core
      PASS |      0 | mpich OSU allgather mpiexec one_core
      PASS |      0 | mpich OSU allgatherv mpiexec one_core
      PASS |      0 | mpich OSU allreduce mpiexec one_core
      PASS |      0 | mpich OSU alltoall mpiexec one_core
      PASS |      0 | mpich OSU alltoallv mpiexec one_core
      PASS |      0 | mpich OSU barrier mpiexec one_core
      PASS |      0 | mpich OSU bcast mpiexec one_core
      PASS |      0 | mpich OSU bibw mpiexec one_core
      PASS |      0 | mpich OSU bw mpiexec one_core
      PASS |      0 | mpich OSU cas_latency mpiexec one_core
      PASS |      0 | mpich OSU fop_latency mpiexec one_core
      PASS |      0 | mpich OSU gather mpiexec one_core
      PASS |      0 | mpich OSU gatherv mpiexec one_core
      PASS |      0 | mpich OSU get_acc_latency mpiexec one_core
      PASS |      0 | mpich OSU get_bw mpiexec one_core
      PASS |      0 | mpich OSU get_latency mpiexec one_core
      PASS |      0 | mpich OSU hello mpiexec one_core
      PASS |      0 | mpich OSU iallgather mpiexec one_core
      PASS |      0 | mpich OSU iallgatherv mpiexec one_core
      PASS |      0 | mpich OSU iallreduce mpiexec one_core
      PASS |      0 | mpich OSU ialltoall mpiexec one_core
      PASS |      0 | mpich OSU ialltoallv mpiexec one_core
      PASS |      0 | mpich OSU ialltoallw mpiexec one_core
      PASS |      0 | mpich OSU ibarrier mpiexec one_core
      PASS |      0 | mpich OSU ibcast mpiexec one_core
      PASS |      0 | mpich OSU igather mpiexec one_core
      PASS |      0 | mpich OSU igatherv mpiexec one_core
      PASS |      0 | mpich OSU init mpiexec one_core
      PASS |      0 | mpich OSU ireduce mpiexec one_core
      PASS |      0 | mpich OSU iscatter mpiexec one_core
      PASS |      0 | mpich OSU iscatterv mpiexec one_core
      PASS |      0 | mpich OSU latency mpiexec one_core
      PASS |      0 | mpich OSU mbw_mr mpiexec one_core
      PASS |      0 | mpich OSU multi_lat mpiexec one_core
      PASS |      0 | mpich OSU put_bibw mpiexec one_core
      PASS |      0 | mpich OSU put_bw mpiexec one_core
      PASS |      0 | mpich OSU put_latency mpiexec one_core
      PASS |      0 | mpich OSU reduce mpiexec one_core
      PASS |      0 | mpich OSU reduce_scatter mpiexec one_core
      PASS |      0 | mpich OSU scatter mpiexec one_core
      PASS |      0 | mpich OSU scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPong mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPing mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Sendrecv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Exchange mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Bcast mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoall mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoallv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allreduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Barrier mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-EXT Window mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibcast mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatterv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoall mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoallv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallreduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibarrier mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_all_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Fetch_and_op mpiexec one_core
      PASS |      0 | mpich IMB-RMA Compare_and_swap mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_all_local mpiexec one_core

Checking for failures and known issues:
  no test failures

Comment 17 Afom T. Michael 2021-01-06 21:59:25 UTC

Moving to verified as test passed. Below shows a sample of test result.

[root@rdma-perf-03 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
[root@rdma-perf-03 ~]$ uname -r
4.18.0-268.el8.x86_64
[root@rdma-perf-03 ~]$ rpm -qa | grep -E "rdma|hwloc|openmpi|mpich|ucx|infiniband" | grep -v "kernel-kernel"
librdmacm-utils-32.0-3.el8.x86_64
mpich-3.3.2-9.el8.x86_64
rdma-core-32.0-3.el8.x86_64
infiniband-diags-32.0-3.el8.x86_64
mpitests-mpich-5.7-1.el8.x86_64
ucx-1.9.0-1.el8.x86_64
rdma-core-devel-32.0-3.el8.x86_64
librdmacm-32.0-3.el8.x86_64
hwloc-libs-2.2.0-1.el8.x86_64
[root@rdma-perf-03 ~]$ ibstatus 
Infiniband device 'mlx5_0' port 1 status:
	default gid:	 fe80:0000:0000:0000:248a:0703:00a3:196c
	base lid:	 0xa
	sm lid:		 0xd
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 100 Gb/sec (4X EDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_1' port 1 status:
	default gid:	 fe80:0000:0000:0000:268a:07ff:fea3:196d
	base lid:	 0x0
	sm lid:		 0x0
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 100 Gb/sec (4X EDR)
	link_layer:	 Ethernet

[root@rdma-perf-03 ~]$


[...snip...]
dnf install -y mpich mpitests-mpich environment-modules
[...snip...]
Package environment-modules-4.5.2-1.el8.x86_64 is already installed.
Dependencies resolved.
================================================================================
 Package             Arch        Version            Repository             Size
================================================================================
Installing:
 mpich               x86_64      3.3.2-9.el8        beaker-AppStream      1.4 M
 mpitests-mpich      x86_64      5.7-1.el8          beaker-AppStream      890 k

Transaction Summary
================================================================================
Install  2 Packages

Total download size: 2.3 M
Installed size: 10 M
Downloading Packages:
(1/2): mpitests-mpich-5.7-1.el8.x86_64.rpm      7.7 MB/s | 890 kB     00:00
(2/2): mpich-3.3.2-9.el8.x86_64.rpm             6.0 MB/s | 1.4 MB     00:00
--------------------------------------------------------------------------------
Total                                           9.6 MB/s | 2.3 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Installing       : mpich-3.3.2-9.el8.x86_64                               1/2
  Installing       : mpitests-mpich-5.7-1.el8.x86_64                        2/2
  Running scriptlet: mpitests-mpich-5.7-1.el8.x86_64                        2/2
  Verifying        : mpich-3.3.2-9.el8.x86_64                               1/2
  Verifying        : mpitests-mpich-5.7-1.el8.x86_64                        2/2
Installed products updated.

Installed:
  mpich-3.3.2-9.el8.x86_64            mpitests-mpich-5.7-1.el8.x86_64

Complete!
[...snip...]
Test results for mpi/mpich2 on rdma-perf-03:
4.18.0-268.el8.x86_64, rdma-core-32.0-3.el8, mlx5, ib0, & mlx5_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mpich OSU acc_latency mpiexec one_core
      PASS |      0 | mpich OSU allgather mpiexec one_core
      PASS |      0 | mpich OSU allgatherv mpiexec one_core
      PASS |      0 | mpich OSU allreduce mpiexec one_core
      PASS |      0 | mpich OSU alltoall mpiexec one_core
      PASS |      0 | mpich OSU alltoallv mpiexec one_core
      PASS |      0 | mpich OSU barrier mpiexec one_core
      PASS |      0 | mpich OSU bcast mpiexec one_core
      PASS |      0 | mpich OSU bibw mpiexec one_core
      PASS |      0 | mpich OSU bw mpiexec one_core
      PASS |      0 | mpich OSU cas_latency mpiexec one_core
      PASS |      0 | mpich OSU fop_latency mpiexec one_core
      PASS |      0 | mpich OSU gather mpiexec one_core
      PASS |      0 | mpich OSU gatherv mpiexec one_core
      PASS |      0 | mpich OSU get_acc_latency mpiexec one_core
      PASS |      0 | mpich OSU get_bw mpiexec one_core
      PASS |      0 | mpich OSU get_latency mpiexec one_core
      PASS |      0 | mpich OSU hello mpiexec one_core
      PASS |      0 | mpich OSU iallgather mpiexec one_core
      PASS |      0 | mpich OSU iallgatherv mpiexec one_core
      PASS |      0 | mpich OSU iallreduce mpiexec one_core
      PASS |      0 | mpich OSU ialltoall mpiexec one_core
      PASS |      0 | mpich OSU ialltoallv mpiexec one_core
      PASS |      0 | mpich OSU ialltoallw mpiexec one_core
      PASS |      0 | mpich OSU ibarrier mpiexec one_core
      PASS |      0 | mpich OSU ibcast mpiexec one_core
      PASS |      0 | mpich OSU igather mpiexec one_core
      PASS |      0 | mpich OSU igatherv mpiexec one_core
      PASS |      0 | mpich OSU init mpiexec one_core
      PASS |      0 | mpich OSU ireduce mpiexec one_core
      PASS |      0 | mpich OSU iscatter mpiexec one_core
      PASS |      0 | mpich OSU iscatterv mpiexec one_core
      PASS |      0 | mpich OSU latency mpiexec one_core
      PASS |      0 | mpich OSU latency_mp mpiexec one_core
      PASS |      0 | mpich OSU mbw_mr mpiexec one_core
      PASS |      0 | mpich OSU multi_lat mpiexec one_core
      PASS |      0 | mpich OSU put_bibw mpiexec one_core
      PASS |      0 | mpich OSU put_bw mpiexec one_core
      PASS |      0 | mpich OSU put_latency mpiexec one_core
      PASS |      0 | mpich OSU reduce mpiexec one_core
      PASS |      0 | mpich OSU reduce_scatter mpiexec one_core
      PASS |      0 | mpich OSU scatter mpiexec one_core
      PASS |      0 | mpich OSU scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPong mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPing mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Sendrecv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Exchange mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Bcast mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoall mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoallv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allreduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Barrier mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-EXT Window mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibcast mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatterv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoall mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoallv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallreduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibarrier mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_all_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Fetch_and_op mpiexec one_core
      PASS |      0 | mpich IMB-RMA Compare_and_swap mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_all_local mpiexec one_core

Checking for failures and known issues:
  no test failures

Comment 19 errata-xmlrpc 2021-05-18 14:42:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (RDMA stack bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1594