RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1171626 - registerable memory calculation is wrong
Summary: registerable memory calculation is wrong
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: openmpi
Version: 6.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Schmidt
QA Contact: Mike Stowell
URL:
Whiteboard:
Depends On:
Blocks: 1269638 1272558
TreeView+ depends on / blocked
 
Reported: 2014-12-08 09:19 UTC by Götz Waschk
Modified: 2016-05-11 01:23 UTC (History)
4 users (show)

Fixed In Version: openmpi-1.10.1-1.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-11 01:23:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0967 0 normal SHIPPED_LIVE rdma bug fix and enhancement update 2016-05-10 22:57:58 UTC

Description Götz Waschk 2014-12-08 09:19:02 UTC 
Description of problem:
After the upgrade from 6.5 to 6.6, openmpi reports it cannot register enough memory for the Infiniband access.

Version-Release number of selected component (if applicable): openmpi-1.8.1-1.el6.x86_64


How reproducible: always


Steps to Reproduce:
run any MPI job on a machine with Infiniband that uses the mlx4 driver and more than 32GB of RAM, e.g.
/usr/lib64/openmpi/bin/mpirun -np 2 /usr/lib64/openmpi/bin/mpitests-osu_get_bw

Actual results:
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              pax98
  Registerable memory:     32768 MiB
  Total memory:            49106 MiB

Your MPI job will continue, but may be behave poorly and/or hang.
--------------------------------------------------------------------------


Expected results:
no such message

Additional info:

openmpi calculates the amount of registerable memory from the values of two Infiniband driver parameters that are not used anymore:
/sys/module/mlx4_core/parameters/log_mtts_per_seg
and /sys/module/mlx4_core/parameters/log_num_mtt

The calculation is all wrong, as the default value of the Infiniband driver in the RHEL 6.6 kernel is twice the amount of physical memory.

A workaround is to load the mlx5_core, in that case openmpi assumes a newer OFED stack and uses the twice the amount of physical memory value.
Comment 5 Michal Schmidt 2016-01-20 20:13:46 UTC 
If I understand correctly, in this case (where the mlx4 driver has commit "mlx4: Scale size of MTT table with system RAM") the WARNING is a false positive.

The relevant openmpi commit is:

commit e868a3c1ffc1e42704b8d8a80615a26bdcf3a511
Author: Devendar Bureddy <devendar>
Date:   Fri Dec 12 05:46:34 2014 +0200

    OMPI: btl openib: fix max registarable memory caluclation

With this commit the mlx4-specific calculation is not done by default (btl_openib_allow_max_memory_registration is on) so all should be fine.

For 6.8 I'm updating openmpi to v1.10.1, which contains the commit.
Comment 11 errata-xmlrpc 2016-05-11 01:23:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0967.html
Note You need to log in before you can comment on or make changes to this bug.