Bug 1618951 - mpich/openmpi python modules are not loadable if python*-mpich/openmpi are not installed
Summary: mpich/openmpi python modules are not loadable if python*-mpich/openmpi are no...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: python-rpm-generators
Version: rawhide
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Nobody's working on this, feel free to take it
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-18 18:34 UTC by Zbigniew Jędrzejewski-Szmek
Modified: 2020-06-07 13:04 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1618949 0 unspecified CLOSED autogenerated dist() deps seem wrong for mpich/openmpi packages 2021-02-22 00:41:40 UTC

Internal Links: 1618949

Description Zbigniew Jędrzejewski-Szmek 2018-08-18 18:34:35 UTC
Description of problem:
For a module installed in /usr/lib64/pythonX.Y/site-packages/mpich/ to be loadable, *two* things must happen:
1. the user must run 'module load mpi' (or similar)
2. pythonX-mpich must be installed.

Point 2. is necessary because that package contains /usr/lib64/pythonX.Y/site-packages/mpich.pth that modifies sys.path.

But AFAICS, the guidelines don't say anything about requiring pythonX-mpich, so it's easy to create a pythonX-*-mpich subpackage that can be installed and cannot be loaded.

There aren't that many mpich/openmpi python subpackages in existence, so just adding the dep everywhere manually is one option. But it seems like this could be handled using autogenerated requires. Essentially, anything package contains /usr/lib64/pythonX.Y/site-packages/mpich/* would require pythonX-mpich, and analogously for /usr/lib64/pythonX.Y/site-packages/openmpi/* and pythonX-openmpi.

Comment 1 Miro Hrončok 2018-08-19 12:00:07 UTC
IMHO it's up to the packager to make the package works. Having the dep autogenerated would indeed be nice, but I don't consider this a bug (rather a RFE).

Comment 2 Zbigniew Jędrzejewski-Szmek 2018-08-19 15:03:56 UTC
Yes, RFE.

Comment 3 Petr Viktorin 2018-08-20 08:05:20 UTC
FWIW, mpich.pth says:
   import sys, os; s = os.getenv('MPI_PYTHON3_SITEARCH'); s and (s in sys.path or sys.path.append(s))

Is there some reason the module doesn't add to PYTHONPATH directly?

Note that there was talk in Python upstream about eventually removing support for executable code in .pth files, as it slows down interpreter startup (for everything that uses Python, not just the package using it), and some people see it as a security risk. I'm quite interested in mpich's use case, in case that discussion comes up again.

Comment 4 Zbigniew Jędrzejewski-Szmek 2018-08-22 06:11:47 UTC
The module does not know which python version will be used (you load the module first, then run a number of commands, which can call either or even both python versions). Using the .pth file allows the decision whether to add /usr/lib/python3.*/site-packages or /usr/lib/python2.*/site-packages to sys.path to be delayed until the interpreter is running. This was implemented as part of https://pagure.io/packaging-committee/issue/563.

> removing support for executable code in .pth files

That would certainly interfere with this use case. I agree that removing executable code in .pth would make a lot of sense, but I hope that a mechanism which allows extending sys.path in a clean and extensible way is still provided.

I think that rather than removing current functionality in .pth, an effort should be made to simply use some declarative mechanism there, so that in 99.9% of cases where it is not needed, the executable code is not used.

Comment 5 Petr Viktorin 2019-01-07 13:52:21 UTC
Unfortunately, I don't know anyone who wants to work on this. To me, it seems that this use of environment modules is quite non-standard.

Comment 6 Ben Cotton 2019-08-13 16:51:48 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 7 Ben Cotton 2019-08-13 19:37:53 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.


Note You need to log in before you can comment on or make changes to this bug.