Bug 801534 - missing mpi.so library in boost mpi python
missing mpi.so library in boost mpi python
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: boost (Show other bugs)
6.1
Unspecified Unspecified
unspecified Severity medium
: rc
: ---
Assigned To: Petr Machata
qe-baseos-tools
:
Depends On:
Blocks: 1056252
  Show dependency treegraph
 
Reported: 2012-03-08 13:37 EST by Riccardo
Modified: 2014-10-14 02:22 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-10-14 02:22:17 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
A fix (1.49 KB, patch)
2012-03-09 13:36 EST, Petr Machata
no flags Details | Diff
Additional fixes. (6.69 KB, patch)
2014-03-06 15:23 EST, Petr Machata
no flags Details | Diff

  None (edit)
Description Riccardo 2012-03-08 13:37:16 EST
Description of problem:
missing mpi.so library. This library is a part of the boost library and is designed to allow using the boost wrappers to mpi through their python interface.



Version-Release number of selected component (if applicable):
boost > 1.35, and any version of redhat


How reproducible:

create a minimal python script called "test.py" 

and write inside 

import mpi
print mpi.rank

then run it as 
mpirun python test.py


Steps to Reproduce:
1. mpirun --np 4 python test.py
2.
3.
  
Actual results:
error

Expected results:
0 1 2 3

Additional info:
this is correctly packaged in recent ubuntu versions, and i contacted with Denis Arnaud (the library packager) to report the problem so he is aware of it.
Comment 2 Petr Machata 2012-03-08 15:48:31 EST
Hmm, we've been suppressing that module for a long time now (the special code for not packaging it is still present in Fedora to this day).  I believe the only reason has been that cmake harness for it is unfinished or buggy.  I'll take a look at what's wrong and why we are keeping this out.
Comment 3 Petr Machata 2012-03-08 20:22:03 EST
This is the immediate dependency error that the spec file is referring to:

c++ [...] -Wl,-soname,mpi.so -o ../../../lib/mpi.so [...] ../../../lib/libboost_python-mt-d.so.5 [...]

We don't ship the -d libraries at all, so mpi.so ends up with the wrong DT_NEEDED.
Comment 4 Petr Machata 2012-03-09 13:36:47 EST
Created attachment 568974 [details]
A fix

This changes the cmake harness and the mpi.so rule to fix building of mpi.so.
Comment 5 Petr Machata 2012-03-09 19:38:22 EST
For proper support across MPI libraries, we are blocked by bug 801945.  Mpich2 should work, but I don't have it configured at the moment, and mpirun gives me an exception.  At least the import line passes.
Comment 6 Petr Machata 2012-03-09 19:41:50 EST
(In reply to comment #0)
> this is correctly packaged in recent ubuntu versions, and i contacted with
> Denis Arnaud (the library packager) to report the problem so he is aware of it.

About this, do you mean cmake upstream packager, or Fedora packager?  If the latter, then the proper interface for reporting bugs is not a private mail to one of the maintainers, but a ticket in bugzilla.
Comment 7 Riccardo 2012-03-26 12:28:31 EDT
Hello,
      sorry for the delay in answering, i did not see this message before.

I was meaning the Fedora Packager. I tried to signal this bug everywhere, on the boost mailing list on the ubuntu mailing list etc. I signaled it also here but to be honest i did not expect anyone to answer to me since i am not a paying user...so thank you for your attention

concerning the  bug 801945

the solution implemented by Ubuntu is quite clean in this sense:
on ubuntu 11.10

locate mpi.so

gives

/etc/alternatives/libmpi.so
/usr/lib/libboost_mpi.so
/usr/lib/libboost_mpi.so.1.46.1
/usr/lib/libexempi.so.3
/usr/lib/libexempi.so.3.2.1
/usr/lib/libmpi.so
/usr/lib/libmpi.so.0
/usr/lib/libmpi.so.0.0.2
/usr/lib/openmpi/lib/libmpi.so
/usr/lib/openmpi/lib/libmpi.so.0.0.2
/usr/lib/pyshared/python2.6/boost/mpi.so
/usr/lib/pyshared/python2.7/boost/mpi.so
/usr/lib/python2.6/dist-packages/boost/mpi.so
/usr/lib/python2.7/dist-packages/boost/mpi.so


if i look within the last directories:

ls -l /usr/lib/python2.7/dist-packages/boost/
total 724
lrwxrwxrwx 1 root root     44 2011-06-04 16:26 __init__.py -> ../../../../share/pyshared/boost/__init__.py
-rw-r--r-- 1 root root    404 2011-12-02 01:31 __init__.pyc
-rw-r--r-- 1 root root 734144 2011-06-04 16:26 mpi.so

the mpi.so file is "standard" and is correctly compiled even by the bjam.
The problem is that in order to be compatible with openmpi > 1.3 the __init__.py file is fundamental as it contains details on how to load the lib within python:

rrossi@mybook:~$ more /usr/lib/python2.7/dist-packages/boost/__init__.py

gives

import sys
if sys.platform == 'linux2':
    import DLFCN as dl
    flags = sys.getdlopenflags()
    sys.setdlopenflags(dl.RTLD_NOW|dl.RTLD_GLOBAL)
    import mpi
    sys.setdlopenflags(flags)
else:
    import mpi

BOTH mpi.so and __init__.py should be located within a directory boost in the path that contains the python modules.
the idea is that once you are in python
you will do

import boost.mpi

and this will eventually load the __init__.py which has the settings needed to correctly load the mpi.so. Without it it does not work :-(



the problem for me is that the bjam compilation shipped with boost only
creates the mpi.so which is installed together with the other boost libraries and not in a subdirectory
"boost". Furthermore the file __init__.py is not created so the lib can not be imported.
 
I have been trying to report this problem, but honestly i don't know where i should do it so most probably i never contacted the right people. If you have any contact with the boost mantainers please inform them of this ...

let me know if you need any other detail

thank you
Riccardo
Comment 8 RHEL Product and Program Management 2012-07-10 04:10:51 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 9 RHEL Product and Program Management 2012-07-10 19:07:55 EDT
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.
Comment 10 RHEL Product and Program Management 2012-09-07 01:29:36 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unable to address this
request at this time.

Red Hat invites you to ask your support representative to
propose this request, if appropriate, in the next release of
Red Hat Enterprise Linux.
Comment 12 Petr Machata 2014-03-06 15:23:52 EST
Created attachment 871607 [details]
Additional fixes.

This should be applied in addition to attachment 568974 [details].  It patches the spec file and adds wrapper script as Riccardo advises in comment 7.

When testing, please make sure that "ssh" client is available on the machine (or in mock), otherwise "import boost.mpi" will fail with a spew of error messages.
Comment 18 errata-xmlrpc 2014-10-14 02:22:17 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1440.html

Note You need to log in before you can comment on or make changes to this bug.