Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
After the upgrade from 6.5 to 6.6, openmpi reports it cannot register enough memory for the Infiniband access.
Version-Release number of selected component (if applicable): openmpi-1.8.1-1.el6.x86_64
How reproducible: always
Steps to Reproduce:
run any MPI job on a machine with Infiniband that uses the mlx4 driver and more than 32GB of RAM, e.g.
/usr/lib64/openmpi/bin/mpirun -np 2 /usr/lib64/openmpi/bin/mpitests-osu_get_bw
Actual results:
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory. This can cause MPI jobs to
run with erratic performance, hang, and/or crash.
This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered. You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.
See this Open MPI FAQ item for more information on these Linux kernel module
parameters:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Local host: pax98
Registerable memory: 32768 MiB
Total memory: 49106 MiB
Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------
Expected results:
no such message
Additional info:
openmpi calculates the amount of registerable memory from the values of two Infiniband driver parameters that are not used anymore:
/sys/module/mlx4_core/parameters/log_mtts_per_seg
and /sys/module/mlx4_core/parameters/log_num_mtt
The calculation is all wrong, as the default value of the Infiniband driver in the RHEL 6.6 kernel is twice the amount of physical memory.
A workaround is to load the mlx5_core, in that case openmpi assumes a newer OFED stack and uses the twice the amount of physical memory value.
If I understand correctly, in this case (where the mlx4 driver has commit "mlx4: Scale size of MTT table with system RAM") the WARNING is a false positive.
The relevant openmpi commit is:
commit e868a3c1ffc1e42704b8d8a80615a26bdcf3a511
Author: Devendar Bureddy <devendar>
Date: Fri Dec 12 05:46:34 2014 +0200
OMPI: btl openib: fix max registarable memory caluclation
With this commit the mlx4-specific calculation is not done by default (btl_openib_allow_max_memory_registration is on) so all should be fine.
For 6.8 I'm updating openmpi to v1.10.1, which contains the commit.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2016-0967.html
Description of problem: After the upgrade from 6.5 to 6.6, openmpi reports it cannot register enough memory for the Infiniband access. Version-Release number of selected component (if applicable): openmpi-1.8.1-1.el6.x86_64 How reproducible: always Steps to Reproduce: run any MPI job on a machine with Infiniband that uses the mlx4 driver and more than 32GB of RAM, e.g. /usr/lib64/openmpi/bin/mpirun -np 2 /usr/lib64/openmpi/bin/mpitests-osu_get_bw Actual results: WARNING: It appears that your OpenFabrics subsystem is configured to only allow registering part of your physical memory. This can cause MPI jobs to run with erratic performance, hang, and/or crash. This may be caused by your OpenFabrics vendor limiting the amount of physical memory that can be registered. You should investigate the relevant Linux kernel module parameters that control how much physical memory can be registered, and increase them to allow registering all physical memory on your machine. See this Open MPI FAQ item for more information on these Linux kernel module parameters: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages Local host: pax98 Registerable memory: 32768 MiB Total memory: 49106 MiB Your MPI job will continue, but may be behave poorly and/or hang. -------------------------------------------------------------------------- Expected results: no such message Additional info: openmpi calculates the amount of registerable memory from the values of two Infiniband driver parameters that are not used anymore: /sys/module/mlx4_core/parameters/log_mtts_per_seg and /sys/module/mlx4_core/parameters/log_num_mtt The calculation is all wrong, as the default value of the Infiniband driver in the RHEL 6.6 kernel is twice the amount of physical memory. A workaround is to load the mlx5_core, in that case openmpi assumes a newer OFED stack and uses the twice the amount of physical memory value.