Bug 1158864 - split compat-openmpi into subpackages by openmpi version
Summary: split compat-openmpi into subpackages by openmpi version
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: compat-openmpi
Version: 6.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Schmidt
QA Contact: Mike Stowell
Lenka Špačková
URL:
Whiteboard:
: 1159807 (view as bug list)
Depends On:
Blocks: 1269638 1270066
TreeView+ depends on / blocked
 
Reported: 2014-10-30 12:15 UTC by Dave Love
Modified: 2016-05-11 01:23 UTC (History)
7 users (show)

Fixed In Version: compat-openmpi-1.4.3-5.el6
Doc Type: Release Note
Doc Text:
Changes in Open MPI distribution Open MPI is an open source Message Passing Interface implementation. The _compat-openmpi_ package, which provides earlier versions of Open MPI for backward compatibility with previous minor releases of Red Hat Enterprise Linux 6, has been split into several subpackages based on the Open MPI version. The names of the subpackages (and their respective environment module names on the x86_64 architecture) are: * _openmpi-1.4_ (openmpi-1.4-x86_64) * _openmpi-1.4-psm_ (openmpi-1.4-psm-x86_64) * _openmpi-1.5.3_ (compat-openmpi-x86_64, aliased as openmpi-1.5.3-x86_64) * _openmpi-1.5.3-psm_ (compat-openmpi-psm-x86_64, aliased as openmpi-1.5.3-psm-x86_64) * _openmpi-1.5.4_ (openmpi-1.5.4-x86_64) * _openmpi-1.8_ (openmpi-x86_64, aliased as openmpi-1.8-x86_64) The "yum install openmpi" command in Red Hat Enterprise Linux 6.8 installs the _openmpi-1.8_ package for maximum compatibility with Red Hat Enterprise Linux 6.7. A later version of Open MPI is available in the _openmpi-1.10_ package.
Clone Of:
Environment:
Last Closed: 2016-05-11 01:23:10 UTC
Target Upstream Version:


Attachments (Terms of Use)
example session (1.84 KB, text/plain)
2014-10-30 12:15 UTC, Dave Love
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0967 normal SHIPPED_LIVE rdma bug fix and enhancement update 2016-05-10 22:57:58 UTC

Description Dave Love 2014-10-30 12:15:48 UTC
Created attachment 952074 [details]
example session

Description of problem:

RHEL 6.6 updates openmpi 1.5 to openmpi 1.8.  This isn't noted in the list of
package updates, and it isn't at all compatible with the previous version 1.5.

At least:
* existing Fortran programs no longer run;
* mpirun arguments have changed, and common ones at least produce a lot of
  warning noise now;
* MCA components and their configuration/defaults have changed (e.g. paffinity is
  absent).

In mitigation, the update is blocked if you have packaged versions of things
installed from current EPEL or self-built, but this will be a real annoyance
if EPEL packages start being rebuilt against the new version.

An example session is attached, showing the result of the upgrade.  This
is a realistic example, though core binding would normally be defaulted in the MCA
parameters, and is one of the defaults which changed in 1.8.  I haven't debugged
why binding now fails, but I have self-built openmpi-1.8 rpms working OK on our
production system in parallel with 1.6 ones (compatible with 1.5).

Version-Release number of selected component (if applicable):
1.8.1

Comment 2 Dave Love 2014-11-02 20:46:47 UTC
I found that the openmpi change is mentioned in the notes, but apparently
only buried in the RDMA section with no public information about it.

Also, from its changelog, the compat-openmpi package is supposed to
cope with this -- it has the 1.5 libraries.  However, it isn't required by the
1.8 package (so you wouldn't know it was required for compatibility if not
pulled in by another package), it's labelled as version 1.4, and it needs the relevant module loading (which won't be done by existing scripts).
The compat- version number is confusing as I wouldn't expect mixing runtime
components between 1.4 and 1.5 to work.

Comment 5 Michal Schmidt 2015-10-08 07:43:47 UTC
Right, compat-openmpi provides the old versions of openmpi. It should have been documented better. In RHEL 6.8 we may update openmpi once more. Then compat-openmpi would subsume openmpi-1.8 too. I agree it is odd that all the old versions are in a single package whose version number refers to only one of the provided versions. And indeed the fact that they are all installed in the same directory is fragile to say the least. I'm thinking about splitting compat-openmpi into subpackages (compat-openmpi14, compat-openmpi15, compat-openmpi18).
The module names will change, that's inevitable.

So "openmpi-1.8 is an incompatible update" is not a bug, but the peculiar compat-openmpi packaging is. Changing the title accordingly.

Comment 6 Michal Schmidt 2015-10-08 07:47:51 UTC
*** Bug 1159807 has been marked as a duplicate of this bug. ***

Comment 8 Dave Love 2016-01-21 15:26:33 UTC
(In reply to Michal Schmidt from comment #5)
> Right, compat-openmpi provides the old versions of openmpi. It should have
> been documented better. In RHEL 6.8 we may update openmpi once more. 

And break everything on HPC systems _again_?  Fair enough to have a more recent version available, but I thought that was what SCLs were for in the RHEL world, although they pretty much interchangeable with environment modules.

> Then
> compat-openmpi would subsume openmpi-1.8 too. I agree it is odd that all the
> old versions are in a single package whose version number refers to only one
> of the provided versions. And indeed the fact that they are all installed in
> the same directory is fragile to say the least. 

If you mean everything lives under %_libdir/compat-openmpi, it's simply broken.  The mpi libraries, the compiler wrappers, the orte support, and the MCA components all need to be compatible.

> I'm thinking about splitting
> compat-openmpi into subpackages (compat-openmpi14, compat-openmpi15,
> compat-openmpi18).
> The module names will change, that's inevitable.

Does anyone at Red Hat understand HPC generally, and how openmpi works in particular?  I don't mean to sound angry, but it seems no-one does, and someone should.

> So "openmpi-1.8 is an incompatible update" is not a bug, but the peculiar
> compat-openmpi packaging is. Changing the title accordingly.

I consider it a bug if the important stuff on our system fundamentally breaks between point releases without the incompatible change even being mentioned in the release notes.  (You can get some idea from the churn in EPEL as a result, and some of that compounded the problems; we have loads of user-compiled and locally-packaged binaries on a service supporting a whole "research-based" university.)  People run RHEL-ish systems for stability and I thought there was even a promise of binary compatibility.

Comment 9 Doug Ledford 2016-01-21 16:55:35 UTC
(In reply to Dave Love from comment #8)
> (In reply to Michal Schmidt from comment #5)
> > Right, compat-openmpi provides the old versions of openmpi. It should have
> > been documented better. In RHEL 6.8 we may update openmpi once more. 
> 
> And break everything on HPC systems _again_?

Unfortunately, yes.  Although I think we can find a way to minimize it in the future, for this update, it will likely involve some pain.

>  Fair enough to have a more
> recent version available, but I thought that was what SCLs were for in the
> RHEL world, although they pretty much interchangeable with environment
> modules.

OpenMPI is a bit of a red-headed step child in that it isn't just a runtime MPI stack.  Because of the evolving nature of the btls it contains, and the fact that it talks directly to RDMA hardware, the updates to OpenMPI are actually hardware enablement updates.  I know in comment #2 you mention that you finally found a release note about the OpenMPI update but it was buried in the RDMA section of the release notes.  When we first added OpenMPI to RHEL it was to support RDMA networks and clusters.  In truth, as far as Red Hat is concerned, OpenMPI and RDMA are inextricably tied.  And for most of our OpenMPI using customers, this is true in their eyes as well.

That all said, I think we can do a better job on how we handle the updates.

> > Then
> > compat-openmpi would subsume openmpi-1.8 too. I agree it is odd that all the
> > old versions are in a single package whose version number refers to only one
> > of the provided versions. And indeed the fact that they are all installed in
> > the same directory is fragile to say the least. 
> 
> If you mean everything lives under %_libdir/compat-openmpi, it's simply
> broken.  The mpi libraries, the compiler wrappers, the orte support, and the
> MCA components all need to be compatible.

Agreed.  The current compat-openmpi is broken.  We need a complete, separate install tree for each compat version.

> > I'm thinking about splitting
> > compat-openmpi into subpackages (compat-openmpi14, compat-openmpi15,
> > compat-openmpi18).
> > The module names will change, that's inevitable.
> 
> Does anyone at Red Hat understand HPC generally, and how openmpi works in
> particular?  I don't mean to sound angry, but it seems no-one does, and
> someone should.

We do.

Most of our customers use multiple MPIs.  OpenMPI being one (and each release of OpenMPI being separate), mvapich, mvapich2, Intel MPI, and some others.  We provide some of these out of the box, some are added after the fact by the customer.  In almost all cases, because we don't provide a full cluster scheduling setup out of the box, the customers have their own cluster scheduling software setup, and that includes they usually have their own means of controlling which MPI is used for each and every job they run.  As a result, our out of the box MPI selection setup has probably gotten less exposure than someone might expect it to.

> > So "openmpi-1.8 is an incompatible update" is not a bug, but the peculiar
> > compat-openmpi packaging is. Changing the title accordingly.
> 
> I consider it a bug if the important stuff on our system fundamentally
> breaks between point releases without the incompatible change even being
> mentioned in the release notes.  (You can get some idea from the churn in
> EPEL as a result, and some of that compounded the problems; we have loads of
> user-compiled and locally-packaged binaries on a service supporting a whole
> "research-based" university.)  People run RHEL-ish systems for stability and
> I thought there was even a promise of binary compatibility.

The RDMA stack in general does not follow the normal RHEL ABI promise.  This has been true ever since we included the stack.  It evolves at such a pace that this sort of ABI promise simply isn't possible.  Since all of the MPIs we ship (with the exception of mpich/mpich2, which are only in rhel7 and not rhel6) are directly tied to our RDMA support, they follow the same rule and do not have an active ABI guarantee.  We *try* to preserve ABI, we wouldn't even have the broken attempt in compat-openmpi otherwise, but it isn't guaranteed.

The real question is how can we make things better for you and still keep our other RDMA customers that expect the latest release on a treadmill happy.

I'm inclined to say that Michal's suggestion of subpackages are the best way to go.  And I would go further and say the parent openmpi package should pull in the subpackages during updates.  And that all packages, including the current package, should use versioned module files, and that we should have a default module file that is a symlink to the current module file version.  Then the recommendation for customers would be that once you compile against a gversion of openmpi, just record what version you compiled against, and on all future runs of the software, use the versioned module name to get the right mpi.  This would protect you against future upgrades.

Comment 10 Michal Schmidt 2016-01-22 14:29:49 UTC
(In reply to Doug Ledford from comment #9)
> (In reply to Dave Love from comment #8)
> > If you mean everything lives under %_libdir/compat-openmpi, it's simply
> > broken.  The mpi libraries, the compiler wrappers, the orte support, and the
> > MCA components all need to be compatible.
> 
> Agreed.  The current compat-openmpi is broken.  We need a complete, separate
> install tree for each compat version.

Yes. I separated them already in compat-openmpi-1.4.3-3.el6.

> ...
> I'm inclined to say that Michal's suggestion of subpackages are the best way
> to go.  And I would go further and say the parent openmpi package should
> pull in the subpackages during updates.

I believe I have already taken care of this by using Obsoletes directives, so that for example upgrading from openmpi-1.5.4 (in RHEL 6.3/6.4/6.5) should install both openmpi-1.10.1 and compat-openmpi15-1.5.4.

>  And that all packages, including
> the current package, should use versioned module files, and that we should
> have a default module file that is a symlink to the current module file
> version. Then the recommendation for customers would be that once you
> compile against a gversion of openmpi, just record what version you compiled
> against, and on all future runs of the software, use the versioned module
> name to get the right mpi.  This would protect you against future upgrades.

So in a RHEL X.Y release "ls -l /etc/modulefiles/mpi" would look something like this:
-rw-r--r--. 1 root root ... openmpi-1.4-x86_64
-rw-r--r--. 1 root root ... openmpi-1.5-x86_64
-rw-r--r--. 1 root root ... openmpi-1.8-x86_64
lrwxrwxrwx. 1 root root ... openmpi-x86_64 -> openmpi-1.8-x86_64

then in RHEL X.(Y+1) it might look like this:
-rw-r--r--. 1 root root ... openmpi-1.10-x86_64
-rw-r--r--. 1 root root ... openmpi-1.4-x86_64
-rw-r--r--. 1 root root ... openmpi-1.5-x86_64
-rw-r--r--. 1 root root ... openmpi-1.8-x86_64
lrwxrwxrwx. 1 root root ... openmpi-x86_64 -> openmpi-1.10-x86_64

I like this.

The only remaining question is what to do in RHEL 6.8 specifically.
RHEL 6.7 obviously did not use this module naming scheme, so customers that currently depend on openmpi-1.8 use its module name "openmpi-x86_64".

There will be an openmpi-1.10 in RHEL 6.8. Let's say it will provide a module called "openmpi-1.10-x86_64". But should it alias the module name "openmpi-x86_64" to itself, or should this name remain dedicated to openmpi-1.8 for the rest of RHEL 6 life?

Comment 11 Michal Schmidt 2016-02-02 12:21:33 UTC
So for RHEL-6.8 I decided to go with the more backwards compatible approach.

First, a summary of what was shipped as the "openmpi" package in each RHEL-6 release:
openmpi-1.4.1-4.3.el6  RHEL-6.0
openmpi-1.4.3-1.1.el6  RHEL-6.1
openmpi-1.5.3-3.el6    RHEL-6.2
openmpi-1.5.4-1.el6    RHEL-6.3, RHEL-6.4
openmpi-1.5.4-2.el6    RHEL-6.5
openmpi-1.8.1-1.el6    RHEL-6.6, RHEL-6.7

The versions up to and including 1.5.3 also *-psm variants.

The "compat-openmpi" package in RHEL-6.7 was a chimera of openmpi v1.4.3 and v1.5.3. Only the v1.5.3 parts had any chance of working, because during the build process it overwrote many of v1.4.3's files.

In RHEL-6.8 the compat-openmpi-1.4.3-5.el6 source package will generate the following binary packages:
openmpi-1.4-1.4.3-5.el6
openmpi-1.4-devel-1.4.3-5.el6
openmpi-1.4-psm-1.4.3-5.el6
openmpi-1.4-psm-devel-1.4.3-5.el6
openmpi-1.5.3-1.5.3-5.el6
openmpi-1.5.3-devel-1.5.3-5.el6
openmpi-1.5.3-psm-1.5.3-5.el6
openmpi-1.5.3-psm-devel-1.5.3-5.el6
openmpi-1.5.4-1.5.4-5.el6
openmpi-1.5.4-devel-1.5.4-5.el6
openmpi-1.8-1.8.1-5.el6
openmpi-1.8-devel-1.8.1-5.el6

where openmpi-1.4 and openmpi-1.5.4 will use directory names /etc/openmpi-$VERSION/, /usr/lib64/openmpi-$VERSION-x86_64 and module names "openmpi-$VERSION-x86_64"

openmpi-1.5.3 will keep using the directory names /etc/compat-openmpi-x86_64, /usr/lib64/compat-openmpi and so on, and the module name "compat-openmpi-x86_64", to maintain compatibility with what was shipped as compat-openmpi in RHEL-6.7.
The module file will be symlinked as "openmpi-1.5.3-x86_64".

openmpi-1.8 will keep using the directory names /etc/openmpi-x86_64, /usr/lib64/openmpi and so on, and the module name "openmpi-x86_64", to maintain compatibility with openmpi from RHEL-6.6 & 6.7.
The module file will be symlinked as "openmpi-1.8-x86_64".
openmpi-1.8-devel Provides "openmpi-devel".

The source package openmpi-1.10.2-2.el6 will generate binary packages:
openmpi-1.10-1.10.2-2.el6
openmpi-1.10-devel-1.10.2-2.el6

where openmpi-1.10 will ship files under /etc/openmpi-1.10-x86_64, /usr/lib64/openmpi-1.10 and so on, and the module "openmpi-1.10-x86_64".

Therefore upgrading from RHEL6.7 should not break anything for openmpi-1.8.1 users and switching to v1.10 will be entirely opt-in.


For RHEL 7 I'd like to switch to the forward-compatible scheme where the directory and module names "openmpi-$VERSION", "openmpi-$VERSION-x86_64" are used consistently by all openmpi versions, and the module name "openmpi-x86_64" is always assigned to the latest openmpi version (or maybe controlled with the alternatives mechanism).

Comment 12 Doug Ledford 2016-02-04 15:17:32 UTC
Sounds reasonable to me.

Comment 19 errata-xmlrpc 2016-05-11 01:23:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0967.html


Note You need to log in before you can comment on or make changes to this bug.