Bug 1171626
| Summary: | registerable memory calculation is wrong | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Götz Waschk <goetz.waschk> |
| Component: | openmpi | Assignee: | Michal Schmidt <mschmidt> |
| Status: | CLOSED ERRATA | QA Contact: | Mike Stowell <mstowell> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.6 | CC: | jshortt, mstowell, wgomerin, zguo |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openmpi-1.10.1-1.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-05-11 01:23:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1269638, 1272558 | ||
If I understand correctly, in this case (where the mlx4 driver has commit "mlx4: Scale size of MTT table with system RAM") the WARNING is a false positive.
The relevant openmpi commit is:
commit e868a3c1ffc1e42704b8d8a80615a26bdcf3a511
Author: Devendar Bureddy <devendar>
Date: Fri Dec 12 05:46:34 2014 +0200
OMPI: btl openib: fix max registarable memory caluclation
With this commit the mlx4-specific calculation is not done by default (btl_openib_allow_max_memory_registration is on) so all should be fine.
For 6.8 I'm updating openmpi to v1.10.1, which contains the commit.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0967.html |
Description of problem: After the upgrade from 6.5 to 6.6, openmpi reports it cannot register enough memory for the Infiniband access. Version-Release number of selected component (if applicable): openmpi-1.8.1-1.el6.x86_64 How reproducible: always Steps to Reproduce: run any MPI job on a machine with Infiniband that uses the mlx4 driver and more than 32GB of RAM, e.g. /usr/lib64/openmpi/bin/mpirun -np 2 /usr/lib64/openmpi/bin/mpitests-osu_get_bw Actual results: WARNING: It appears that your OpenFabrics subsystem is configured to only allow registering part of your physical memory. This can cause MPI jobs to run with erratic performance, hang, and/or crash. This may be caused by your OpenFabrics vendor limiting the amount of physical memory that can be registered. You should investigate the relevant Linux kernel module parameters that control how much physical memory can be registered, and increase them to allow registering all physical memory on your machine. See this Open MPI FAQ item for more information on these Linux kernel module parameters: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages Local host: pax98 Registerable memory: 32768 MiB Total memory: 49106 MiB Your MPI job will continue, but may be behave poorly and/or hang. -------------------------------------------------------------------------- Expected results: no such message Additional info: openmpi calculates the amount of registerable memory from the values of two Infiniband driver parameters that are not used anymore: /sys/module/mlx4_core/parameters/log_mtts_per_seg and /sys/module/mlx4_core/parameters/log_num_mtt The calculation is all wrong, as the default value of the Infiniband driver in the RHEL 6.6 kernel is twice the amount of physical memory. A workaround is to load the mlx5_core, in that case openmpi assumes a newer OFED stack and uses the twice the amount of physical memory value.