Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1900884

Summary:	[RHEL8.4] mvapich2 OSU latency_mp benchmark times out when run with "mpirun_rsh" on MLX4 IB0 devices
Product:	Red Hat Enterprise Linux 8	Reporter:	Brian Chae <bchae>
Component:	mpitests	Assignee:	Honggang LI <honli>
Status:	CLOSED NOTABUG	QA Contact:	Brian Chae <bchae>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	cwei, rdma-dev-team, tmichael
Target Milestone:	rc	Keywords:	Triaged
Target Release:	8.4	Flags:	pm-rhel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-15 14:35:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1903942

Description Brian Chae 2020-11-23 22:13:24 UTC

Description of problem:

When the mvapich2 benchmarks run with "mpirun_rsh" on MLX4 IB0 devices, only one fails with timing out - mvapich2 OSU latency_mp. This particular benchmark must be part of new mpitests-mvapich2 benchmark set, which is not part of mpitests-mvapich2-5.6.2-1.el8.x86_64 in RHEL8.3 builds.


Version-Release number of selected component (if applicable):

DISTRO=RHEL-8.4.0-20201117.n.0

+ [20-11-23 13:11:13] cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)

+ [20-11-23 13:11:13] uname -a
Linux rdma-perf-01.lab.bos.redhat.com 4.18.0-249.el8.x86_64 #1 SMP Thu Nov 12 05:15:48 EST 2020 x86_64 x86_64 x86_64 GNU/Linux

+ [20-11-23 13:11:13] cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-249.el8.x86_64 root=/dev/mapper/rhel_rdma--perf--01-root ro intel_idle.max_cstate=0 intremap=no_x2apic_optout processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH intel_idle.max_cstate=0 intremap=no_x2apic_optout processor.max_cstate=0 crashkernel=auto resume=/dev/mapper/rhel_rdma--perf--01-swap rd.lvm.lv=rhel_rdma-perf-01/root rd.lvm.lv=rhel_rdma-perf-01/swap console=ttyS1,115200n81

+ [20-11-23 13:11:13] rpm -q rdma-core linux-firmware
rdma-core-32.0-1.el8.x86_64

linux-firmware-20201022-100.gitdae4b4cd.el8.noarch
+ [20-11-23 13:11:13] tail /sys/class/infiniband/mlx4_0/fw_ver
2.42.5000

+ [20-11-23 13:11:13] lspci
+ [20-11-23 13:11:13] grep -i -e ethernet -e infiniband -e omni -e ConnectX
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]


Installed:
  mpitests-mvapich2-5.6.3-1.el8.x86_64        mvapich2-2.3.4-1.el8.x86_64       



How reproducible:

100%

Steps to Reproduce:
1. Install the above software onto MLX4 ConnectX-3 equipped client/server hosts
2. Run the mvapich2 benchmarks on the client, shown as below

[root@rdma-perf-01 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun_rsh -hostfile /root/hfile_one_core -np 2 /usr/lib64/mvapich2/bin/mpitests-osu_latency_mp


3.

Actual results:

After 3 minutes, the command will time out with the following message:

# OSU MPI Multi-process Latency Test v5.6.3
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# Size          Latency (us)

[rdma-perf-01.lab.bos.redhat.com:mpirun_rsh][signal_processor] Caught signal 15, killing job
[rdma-perf-01.lab.bos.redhat.com:mpirun_rsh][signal_processor] Caught signal 15, killing job
[root@rdma-perf-01 ~]$ [rdma-perf-00.lab.bos.redhat.com:mpispawn_0][report_error] connect() failed: Connection refused (111)
[rdma-perf-01.lab.bos.redhat.com:mpispawn_1][read_size] Unexpected End-Of-File on file descriptor 5. MPI process died?
[rdma-perf-01.lab.bos.redhat.com:mpispawn_1][read_size] Unexpected End-Of-File on file descriptor 5. MPI process died?
[rdma-perf-01.lab.bos.redhat.com:mpispawn_1][handle_mt_peer] Error while reading PMI socket. MPI process died?
[rdma-perf-01.lab.bos.redhat.com:mpispawn_1][report_error] connect() failed: Connection refused (111)





Expected results:

Normal OSU benchmark output showing the data size and measured latency values.


Additional info:

Also, when the same osu_latency_mp benchmark is run with "mpirun" command, there is no output from the run, but, only the header info only shown, as below.


+ [20-11-23 13:27:23] timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 /usr/lib64/mvapich2/bin/mpitests-osu_latency_mp
# OSU MPI Multi-process Latency Test v5.6.3
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# Size          Latency (us) <<<================================= NO OUTPUT
+ [20-11-23 13:30:23] __MPI_check_result 0 mpitests-mvapich2 OSU /usr/lib64/mvapich2/bin/mpitests-osu_latency_mp mpirun /root/hfile_one_core
+ [20-11-23 13:30:23] '[' 6 -ne 6 ']'
+ [20-11-23 13:30:23] local status=0
+ [20-11-23 13:30:23] local pkg=mvapich2
+ [20-11-23 13:30:23] local benchmark=OSU
++ [20-11-23 13:30:23] basename /usr/lib64/mvapich2/bin/mpitests-osu_latency_mp
+ [20-11-23 13:30:23] local app=mpitests-osu_latency_mp

Comment 3 Honggang LI 2020-12-07 02:02:22 UTC

./osu-micro-benchmarks-5.6.3/mpi/pt2pt/osu_latency_mp.c is a new benchmark. First release in osu benchmark release 5.6.3 .
 So change the component to 'mpitests'.

Comment 4 Honggang LI 2020-12-15 09:19:32 UTC

The benchmark 'osu_latency_mp' uses system call 'fork'. The libibverbs library is not fork safe by default.
So, program calls 'fork' should call 'ibv_fork_init'. 

Setting the environment variable RDMAV_FORK_SAFE or IBV_FORK_SAFE has the same effect as calling ibv_fork_init().

[root@rdma-dev-10 ~]$ mpirun -genv RDMAV_FORK_SAFE 1 -hostfile ./one_core -np 2 mpitests-osu_latency_mp 
# OSU MPI Multi-process Latency Test v5.6.3
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# Size          Latency (us)
0                       1.38
1                       1.47
2                       1.45
4                       1.44
8                       1.44
16                      1.32
32                      1.15
64                      1.09
128                     1.16
256                     1.81
512                     1.99
1024                    2.31
2048                    2.95
4096                    4.15
8192                    5.66
16384                   7.19
32768                   9.81
65536                  14.84
131072                 25.21
262144                 46.24
524288                 88.01
1048576               171.53
2097152               338.79
4194304               673.63

[root@rdma-dev-10 ~]$ mpirun -genv IBV_FORK_SAFE 1 -hostfile ./one_core -np 2 mpitests-osu_latency_mp 
# OSU MPI Multi-process Latency Test v5.6.3
# Number of forked processes in sender: 2
# Number of forked processes in receiver: 2
# Size          Latency (us)
0                       1.39
1                       1.44
2                       1.44
4                       1.44
8                       1.21
16                      1.05
32                      1.02
64                      1.09
128                     1.16
256                     1.82
512                     1.98
1024                    2.31
2048                    2.96
4096                    4.14
8192                    5.66
16384                   7.28
32768                   9.70
65536                  14.93
131072                 25.23
262144                 46.17
524288                 87.99
1048576               171.44
2097152               338.78
4194304               673.45

Comment 6 Honggang LI 2020-12-15 14:35:32 UTC

Close this bug as NOTABUG. Please see upstream feedback for details.

<snip>
Please note that this was intentional. The purpose of this test was to check if the underlying IB-enabled MPI communication runtime has taken care of fork safety even if the application has not.

We noticed that when using frameworks like TensorFlow 2.0 and higher over Horovod+MPI, the training would hang because TensorFlow was using fork. To simulate use cases like this and ensure MPI libraries will not hang, we had created this test so that we can catch this internally in our testing and validation.

We also introduced a new variable "MV2_SUPPORT_FORK_SAFETY" (which is disabled by default due to performance reasons) to make sure MVAPICH2 takes care of fork safety for applications that require it.
<snip>