Bug 1618951

Summary:	mpich/openmpi python modules are not loadable if python*-mpich/openmpi are not installed
Product:	[Fedora] Fedora	Reporter:	Zbigniew Jędrzejewski-Szmek <zbyszek>
Component:	python-rpm-generators	Assignee:	Nobody's working on this, feel free to take it <nobody>
Status:	NEW ---	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	low
Version:	rawhide	CC:	cstratak, lbalhar, mcyprian, mhroncok, pviktori, shcherbina.iryna, torsava
Target Milestone:	---	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Zbigniew Jędrzejewski-Szmek 2018-08-18 18:34:35 UTC

Description of problem:
For a module installed in /usr/lib64/pythonX.Y/site-packages/mpich/ to be loadable, *two* things must happen:
1. the user must run 'module load mpi' (or similar)
2. pythonX-mpich must be installed.

Point 2. is necessary because that package contains /usr/lib64/pythonX.Y/site-packages/mpich.pth that modifies sys.path.

But AFAICS, the guidelines don't say anything about requiring pythonX-mpich, so it's easy to create a pythonX-*-mpich subpackage that can be installed and cannot be loaded.

There aren't that many mpich/openmpi python subpackages in existence, so just adding the dep everywhere manually is one option. But it seems like this could be handled using autogenerated requires. Essentially, anything package contains /usr/lib64/pythonX.Y/site-packages/mpich/* would require pythonX-mpich, and analogously for /usr/lib64/pythonX.Y/site-packages/openmpi/* and pythonX-openmpi.

Comment 1 Miro Hrončok 2018-08-19 12:00:07 UTC

IMHO it's up to the packager to make the package works. Having the dep autogenerated would indeed be nice, but I don't consider this a bug (rather a RFE).

Comment 2 Zbigniew Jędrzejewski-Szmek 2018-08-19 15:03:56 UTC

Yes, RFE.

Comment 3 Petr Viktorin (pviktori) 2018-08-20 08:05:20 UTC

FWIW, mpich.pth says:
   import sys, os; s = os.getenv('MPI_PYTHON3_SITEARCH'); s and (s in sys.path or sys.path.append(s))

Is there some reason the module doesn't add to PYTHONPATH directly?

Note that there was talk in Python upstream about eventually removing support for executable code in .pth files, as it slows down interpreter startup (for everything that uses Python, not just the package using it), and some people see it as a security risk. I'm quite interested in mpich's use case, in case that discussion comes up again.

Comment 4 Zbigniew Jędrzejewski-Szmek 2018-08-22 06:11:47 UTC

The module does not know which python version will be used (you load the module first, then run a number of commands, which can call either or even both python versions). Using the .pth file allows the decision whether to add /usr/lib/python3.*/site-packages or /usr/lib/python2.*/site-packages to sys.path to be delayed until the interpreter is running. This was implemented as part of https://pagure.io/packaging-committee/issue/563.

> removing support for executable code in .pth files

That would certainly interfere with this use case. I agree that removing executable code in .pth would make a lot of sense, but I hope that a mechanism which allows extending sys.path in a clean and extensible way is still provided.

I think that rather than removing current functionality in .pth, an effort should be made to simply use some declarative mechanism there, so that in 99.9% of cases where it is not needed, the executable code is not used.

Comment 5 Petr Viktorin (pviktori) 2019-01-07 13:52:21 UTC

Unfortunately, I don't know anyone who wants to work on this. To me, it seems that this use of environment modules is quite non-standard.

Comment 6 Ben Cotton 2019-08-13 16:51:48 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 7 Ben Cotton 2019-08-13 19:37:53 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.