Bug 2212010

Summary: [RHEL-9.3] Update mpich package
Product: Red Hat Enterprise Linux 9 Reporter: Kamal Heib <kheib>
Component: mpichAssignee: Kamal Heib <kheib>
Status: VERIFIED --- QA Contact: Afom T. Michael <tmichael>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 9.3CC: hwkernel-mgr, rdma-dev-team, tmichael
Target Milestone: rcKeywords: Triaged
Target Release: 9.3   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: mpich-4.1.1-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Component Upgrade
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kamal Heib 2023-06-03 00:08:04 UTC
Description of problem:
Update mpich for rhel-9.3 release 

Version-Release number of selected component (if applicable):
4.1.1

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Afom T. Michael 2023-06-10 23:56:15 UTC
Marking as "Verified: Tested" as tests with mpich-4.1.1-1.el9 passed.

[root@rdma-qe-41 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.3 Beta (Plow)
[root@rdma-qe-41 ~]$ uname -r
5.14.0-324.el9.x86_64
[root@rdma-qe-41 ~]$ grep DISTRO /etc/motd | uniq | tr -d " "
DISTRO=RHEL-9.3.0-20230610.33
[root@rdma-qe-41 ~]$ rpm -q rdma-core ucx 
rdma-core-46.0-1.el9.x86_64
ucx-1.14.1-1.el9.1.x86_64
[root@rdma-qe-41 ~]$ lspci | grep Mellanox
c4:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
c4:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
[root@rdma-qe-41 ~]$ ibstatus 
Infiniband device 'mlx5_0' port 1 status:
	default gid:	 fe80:0000:0000:0000:946d:ae03:0041:8012
	base lid:	 0x43
	sm lid:		 0x3
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 100 Gb/sec (4X EDR)
	link_layer:	 InfiniBand

Infiniband device 'mlx5_1' port 1 status:
	default gid:	 fe80:0000:0000:0000:966d:aeff:fe41:8013
	base lid:	 0x0
	sm lid:		 0x0
	state:		 4: ACTIVE
	phys state:	 5: LinkUp
	rate:		 100 Gb/sec (4X EDR)
	link_layer:	 Ethernet

[root@rdma-qe-41 ~]$ 

[...snip...]
Installed:
  mpich-4.1.1-1.el9.x86_64              mpich-autoload-4.1.1-1.el9.x86_64       
  mpich-devel-4.1.1-1.el9.x86_64        mpitests-mpich-5.8-1.el9.x86_64      
[...snip...]

mpi/mpich2 test results on rdma-qe-40/rdma-qe-41 & Beaker job J:7960273:
5.14.0-324.el9.x86_64, rdma-core-46.0-1.el9, mlx5, ib0, ConnectX-7 & mlx5_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mpich OSU acc_latency mpiexec one_core
      PASS |      0 | mpich OSU allgather mpiexec one_core
      PASS |      0 | mpich OSU allgatherv mpiexec one_core
      PASS |      0 | mpich OSU allreduce mpiexec one_core
      PASS |      0 | mpich OSU alltoall mpiexec one_core
      PASS |      0 | mpich OSU alltoallv mpiexec one_core
      PASS |      0 | mpich OSU barrier mpiexec one_core
      PASS |      0 | mpich OSU bcast mpiexec one_core
      PASS |      0 | mpich OSU bibw mpiexec one_core
      PASS |      0 | mpich OSU bw mpiexec one_core
      PASS |      0 | mpich OSU cas_latency mpiexec one_core
      PASS |      0 | mpich OSU fop_latency mpiexec one_core
      PASS |      0 | mpich OSU gather mpiexec one_core
      PASS |      0 | mpich OSU gatherv mpiexec one_core
      PASS |      0 | mpich OSU get_acc_latency mpiexec one_core
      PASS |      0 | mpich OSU get_bw mpiexec one_core
      PASS |      0 | mpich OSU get_latency mpiexec one_core
      PASS |      0 | mpich OSU hello mpiexec one_core
      PASS |      0 | mpich OSU iallgather mpiexec one_core
      PASS |      0 | mpich OSU iallgatherv mpiexec one_core
      PASS |      0 | mpich OSU iallreduce mpiexec one_core
      PASS |      0 | mpich OSU ialltoall mpiexec one_core
      PASS |      0 | mpich OSU ialltoallv mpiexec one_core
      PASS |      0 | mpich OSU ialltoallw mpiexec one_core
      PASS |      0 | mpich OSU ibarrier mpiexec one_core
      PASS |      0 | mpich OSU ibcast mpiexec one_core
      PASS |      0 | mpich OSU igather mpiexec one_core
      PASS |      0 | mpich OSU igatherv mpiexec one_core
      PASS |      0 | mpich OSU init mpiexec one_core
      PASS |      0 | mpich OSU ireduce mpiexec one_core
      PASS |      0 | mpich OSU iscatter mpiexec one_core
      PASS |      0 | mpich OSU iscatterv mpiexec one_core
      PASS |      0 | mpich OSU latency mpiexec one_core
      PASS |      0 | mpich OSU latency_mp mpiexec one_core
      PASS |      0 | mpich OSU mbw_mr mpiexec one_core
      PASS |      0 | mpich OSU multi_lat mpiexec one_core
      PASS |      0 | mpich OSU put_bibw mpiexec one_core
      PASS |      0 | mpich OSU put_bw mpiexec one_core
      PASS |      0 | mpich OSU put_latency mpiexec one_core
      PASS |      0 | mpich OSU reduce mpiexec one_core
      PASS |      0 | mpich OSU reduce_scatter mpiexec one_core
      PASS |      0 | mpich OSU scatter mpiexec one_core
      PASS |      0 | mpich OSU scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPong mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPing mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Sendrecv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Exchange mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Bcast mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoall mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoallv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allreduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Barrier mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-EXT Window mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibcast mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatterv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoall mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoallv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallreduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibarrier mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_all_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Fetch_and_op mpiexec one_core
      PASS |      0 | mpich IMB-RMA Compare_and_swap mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_all_local mpiexec one_core
Checking for failures and known issues:
  no test failures

Comment 4 Afom T. Michael 2023-06-29 04:54:25 UTC
RHEL-9.3 has mpich-4.1.1-1.el9.x86_64 and tests have passed. Therefore, marking this as 'Verified'.

Installed:
  mpich-4.1.1-1.el9.x86_64            mpitests-mpich-5.8-1.el9.x86_64  

mpi-mpich2 test results on rdma-perf-02/rdma-perf-03 & Beaker job J:8017687:
5.14.0-332.el9.x86_64, rdma-core-46.0-1.el9, mlx5, ib0, ConnectX-5 & mlx5_0
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mpich OSU acc_latency mpiexec one_core
      PASS |      0 | mpich OSU allgather mpiexec one_core
      PASS |      0 | mpich OSU allgatherv mpiexec one_core
      PASS |      0 | mpich OSU allreduce mpiexec one_core
      PASS |      0 | mpich OSU alltoall mpiexec one_core
      PASS |      0 | mpich OSU alltoallv mpiexec one_core
      PASS |      0 | mpich OSU barrier mpiexec one_core
      PASS |      0 | mpich OSU bcast mpiexec one_core
      PASS |      0 | mpich OSU bibw mpiexec one_core
      PASS |      0 | mpich OSU bw mpiexec one_core
      PASS |      0 | mpich OSU cas_latency mpiexec one_core
      PASS |      0 | mpich OSU fop_latency mpiexec one_core
      PASS |      0 | mpich OSU gather mpiexec one_core
      PASS |      0 | mpich OSU gatherv mpiexec one_core
      PASS |      0 | mpich OSU get_acc_latency mpiexec one_core
      PASS |      0 | mpich OSU get_bw mpiexec one_core
      PASS |      0 | mpich OSU get_latency mpiexec one_core
      PASS |      0 | mpich OSU hello mpiexec one_core
      PASS |      0 | mpich OSU iallgather mpiexec one_core
      PASS |      0 | mpich OSU iallgatherv mpiexec one_core
      PASS |      0 | mpich OSU iallreduce mpiexec one_core
      PASS |      0 | mpich OSU ialltoall mpiexec one_core
      PASS |      0 | mpich OSU ialltoallv mpiexec one_core
      PASS |      0 | mpich OSU ialltoallw mpiexec one_core
      PASS |      0 | mpich OSU ibarrier mpiexec one_core
      PASS |      0 | mpich OSU ibcast mpiexec one_core
      PASS |      0 | mpich OSU igather mpiexec one_core
      PASS |      0 | mpich OSU igatherv mpiexec one_core
      PASS |      0 | mpich OSU init mpiexec one_core
      PASS |      0 | mpich OSU ireduce mpiexec one_core
      PASS |      0 | mpich OSU iscatter mpiexec one_core
      PASS |      0 | mpich OSU iscatterv mpiexec one_core
      PASS |      0 | mpich OSU latency mpiexec one_core
      PASS |      0 | mpich OSU latency_mp mpiexec one_core
      PASS |      0 | mpich OSU mbw_mr mpiexec one_core
      PASS |      0 | mpich OSU multi_lat mpiexec one_core
      PASS |      0 | mpich OSU put_bibw mpiexec one_core
      PASS |      0 | mpich OSU put_bw mpiexec one_core
      PASS |      0 | mpich OSU put_latency mpiexec one_core
      PASS |      0 | mpich OSU reduce mpiexec one_core
      PASS |      0 | mpich OSU reduce_scatter mpiexec one_core
      PASS |      0 | mpich OSU scatter mpiexec one_core
      PASS |      0 | mpich OSU scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPong mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPing mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Sendrecv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Exchange mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Bcast mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoall mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoallv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allreduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Barrier mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-EXT Window mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibcast mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatterv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoall mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoallv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallreduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibarrier mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_all_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Fetch_and_op mpiexec one_core
      PASS |      0 | mpich IMB-RMA Compare_and_swap mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_all_local mpiexec one_core
Checking for failures and known issues:
  no test failures

mpi-mpich2 test results on rdma-perf-02/rdma-perf-03 & Beaker job J:8017687:
5.14.0-332.el9.x86_64, rdma-core-46.0-1.el9, mlx5, roce.45, ConnectX-5 & mlx5_1
    Result | Status | Test
  ---------+--------+------------------------------------
      PASS |      0 | mpich OSU acc_latency mpiexec one_core
      PASS |      0 | mpich OSU allgather mpiexec one_core
      PASS |      0 | mpich OSU allgatherv mpiexec one_core
      PASS |      0 | mpich OSU allreduce mpiexec one_core
      PASS |      0 | mpich OSU alltoall mpiexec one_core
      PASS |      0 | mpich OSU alltoallv mpiexec one_core
      PASS |      0 | mpich OSU barrier mpiexec one_core
      PASS |      0 | mpich OSU bcast mpiexec one_core
      PASS |      0 | mpich OSU bibw mpiexec one_core
      PASS |      0 | mpich OSU bw mpiexec one_core
      PASS |      0 | mpich OSU cas_latency mpiexec one_core
      PASS |      0 | mpich OSU fop_latency mpiexec one_core
      PASS |      0 | mpich OSU gather mpiexec one_core
      PASS |      0 | mpich OSU gatherv mpiexec one_core
      PASS |      0 | mpich OSU get_acc_latency mpiexec one_core
      PASS |      0 | mpich OSU get_bw mpiexec one_core
      PASS |      0 | mpich OSU get_latency mpiexec one_core
      PASS |      0 | mpich OSU hello mpiexec one_core
      PASS |      0 | mpich OSU iallgather mpiexec one_core
      PASS |      0 | mpich OSU iallgatherv mpiexec one_core
      PASS |      0 | mpich OSU iallreduce mpiexec one_core
      PASS |      0 | mpich OSU ialltoall mpiexec one_core
      PASS |      0 | mpich OSU ialltoallv mpiexec one_core
      PASS |      0 | mpich OSU ialltoallw mpiexec one_core
      PASS |      0 | mpich OSU ibarrier mpiexec one_core
      PASS |      0 | mpich OSU ibcast mpiexec one_core
      PASS |      0 | mpich OSU igather mpiexec one_core
      PASS |      0 | mpich OSU igatherv mpiexec one_core
      PASS |      0 | mpich OSU init mpiexec one_core
      PASS |      0 | mpich OSU ireduce mpiexec one_core
      PASS |      0 | mpich OSU iscatter mpiexec one_core
      PASS |      0 | mpich OSU iscatterv mpiexec one_core
      PASS |      0 | mpich OSU latency mpiexec one_core
      PASS |      0 | mpich OSU latency_mp mpiexec one_core
      PASS |      0 | mpich OSU mbw_mr mpiexec one_core
      PASS |      0 | mpich OSU multi_lat mpiexec one_core
      PASS |      0 | mpich OSU put_bibw mpiexec one_core
      PASS |      0 | mpich OSU put_bw mpiexec one_core
      PASS |      0 | mpich OSU put_latency mpiexec one_core
      PASS |      0 | mpich OSU reduce mpiexec one_core
      PASS |      0 | mpich OSU reduce_scatter mpiexec one_core
      PASS |      0 | mpich OSU scatter mpiexec one_core
      PASS |      0 | mpich OSU scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPong mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 PingPing mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Sendrecv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Exchange mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Bcast mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allgatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gather mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Gatherv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Scatterv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoall mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Alltoallv mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Reduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Allreduce mpiexec one_core
      PASS |      0 | mpich IMB-MPI1 Barrier mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO S_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Write_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO P_Read_priv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_indv mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_expl mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Write_shared mpiexec one_core
      PASS |      0 | mpich IMB-IO C_Read_shared mpiexec one_core
      PASS |      0 | mpich IMB-EXT Window mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Unidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Get mpiexec one_core
      PASS |      0 | mpich IMB-EXT Bidir_Put mpiexec one_core
      PASS |      0 | mpich IMB-EXT Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibcast mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallgatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igather mpiexec one_core
      PASS |      0 | mpich IMB-NBC Igatherv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iscatterv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoall mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ialltoallv mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ireduce_scatter mpiexec one_core
      PASS |      0 | mpich IMB-NBC Iallreduce mpiexec one_core
      PASS |      0 | mpich IMB-NBC Ibarrier mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Unidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Bidir_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA One_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_put_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA All_get_all mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Put_all_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_put mpiexec one_core
      PASS |      0 | mpich IMB-RMA Exchange_get mpiexec one_core
      PASS |      0 | mpich IMB-RMA Accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_accumulate mpiexec one_core
      PASS |      0 | mpich IMB-RMA Fetch_and_op mpiexec one_core
      PASS |      0 | mpich IMB-RMA Compare_and_swap mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_local mpiexec one_core
      PASS |      0 | mpich IMB-RMA Get_all_local mpiexec one_core
Checking for failures and known issues:
  no test failures