Bug 1618949

Summary:	autogenerated dist() deps seem wrong for mpich/openmpi packages
Product:	[Fedora] Fedora	Reporter:	Zbigniew Jędrzejewski-Szmek <zbyszek>
Component:	python-rpm-generators	Assignee:	Neal Gompa <ngompa13>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	cstratak, lbalhar, mcyprian, mhroncok, ngompa13, pviktori, shcherbina.iryna, torsava
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-rpm-generators-6-1.fc29	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-10-07 21:00:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Zbigniew Jędrzejewski-Szmek 2018-08-18 18:01:41 UTC

Description of problem:
We have various python packages that provide mpich and openmpi variants in a separate subpackage. For the ones that have egg-info, the dependency generator creates pythonX.Ydist() and pythonXdist() provides. But those modules are not directly importable in python (they are outside of the normal sys.path), and only become loadable when the specific mpich or openmpi environment module is loaded.

For example mpi4py:
$ dnf repoquery --release=rawhide --whatprovides 'python2.7dist(mpi4py)'
python2-mpi4py-mpich-0:3.0.0-4.fc29.x86_64
python2-mpi4py-openmpi-0:3.0.0-4.fc29.x86_64
$ rpm -q python2-mpi4py-mpich
python2-mpi4py-mpich-2.0.0-14.fc27.x86_64
$ python2 -m mpi4py
/usr/bin/python2: No module named mpi4py
$ module load mpi/mpich-x86_64
$ python2 -m mpi4py
Usage: mpi4py [options] <command> [args]

I don't see why the dependency script would look at the egg-info files outside of usual python path. This makes it impossible to reliably use generated dependencies from other packages.

Version-Release number of selected component (if applicable):
I think this applies across all versions where python provides are automatically generated.

(I have a vague memory of this coming up before, but I can't find any links. Please excuse my short memory.)

Comment 1 Miro Hrončok 2018-08-19 11:58:45 UTC

Neal, could you please have a look?

Comment 2 Neal Gompa 2018-08-20 22:36:16 UTC

Now, feel free for someone to correct me on this, but I *think* this is caused by how the regex actually works:

This is the regex: ^/usr/lib(64)?/python[[:digit:]]\\.[[:digit:]]/.*\\.(dist.*|egg.*)$

Note the ".*" between "python[[:digit:]]\\.[[:digit:]]/" and "\\.(dist.*|egg.*)$".

That means it'd match for every path that fits within, which includes paths like "/usr/lib64/python2.7/site-packages/mpich/mpi4py-3.0.0-py2.7.egg-info".

I'd love to tighten this, but I'm not sure how, as my command of regex isn't that great...

Comment 3 Zbigniew Jędrzejewski-Szmek 2018-08-21 07:57:26 UTC

So there are additional problems with this regexp. I'll match anything that has "dist" and "egg" in it, e.g. /usr/lib/python3.7/eggplant.py, /usr/lib64/python3.6/site-packages/sipdistutils.py.

I'd also simplify it by not using [[:digit:]], because [0-9] is shorter.

I looked for all files containing "dist" or "egg", and apart from the obvious accidental matches like sipdistutils.py, they all have either .dist-info or .egg-info suffix.

I think the regex should be something like this:
^/usr/lib(64)?/python[0-9]\\.[0-9]/[^/]*\\.(dist|egg)-info$

Comment 4 Petr Viktorin (pviktori) 2018-08-21 08:25:42 UTC

Also add a + after [0-9]\.[0-9] so it will match Python 3.10 in a few years.

^/usr/lib(64)?/python[0-9]\\.[0-9]+/[^/]*\\.(dist|egg)-info$

Comment 5 Neal Gompa 2018-08-21 13:12:54 UTC

(In reply to Zbigniew Jędrzejewski-Szmek from comment #3)
> So there are additional problems with this regexp. I'll match anything that
> has "dist" and "egg" in it, e.g. /usr/lib/python3.7/eggplant.py,
> /usr/lib64/python3.6/site-packages/sipdistutils.py.
> 
> I'd also simplify it by not using [[:digit:]], because [0-9] is shorter.
> 
> I looked for all files containing "dist" or "egg", and apart from the
> obvious accidental matches like sipdistutils.py, they all have either
> .dist-info or .egg-info suffix.
> 
> I think the regex should be something like this:
> ^/usr/lib(64)?/python[0-9]\\.[0-9]/[^/]*\\.(dist|egg)-info$

The generator is capable of generating dependencies on .egg and can follow .egg-link files.

Comment 6 Neal Gompa 2018-08-21 13:13:33 UTC

I suppose technically we could read .whl, too... That said, I don't think we want either .egg or .whl files read.

Comment 7 Zbigniew Jędrzejewski-Szmek 2018-08-21 14:32:51 UTC

Right. So the pattern should much whatever specific extensions are wanted, but not more. So something like:

^/usr/lib(64)?/python[0-9]\\.[0-9]/[^/]*\\.(dist-info|egg-info|egg-link)$

(with whatever other items added in the parenthesized section as needed.)

Comment 8 Petr Viktorin (pviktori) 2018-09-24 12:22:36 UTC

Neal, are you working on this? Should we take over?

Comment 9 Neal Gompa 2018-09-26 13:50:35 UTC

I'm working on this, I'll have something proposed in rpm upstream soonish.

Comment 10 Neal Gompa 2018-10-01 21:30:18 UTC

I've proposed a change in upstream RPM for this: https://github.com/rpm-software-management/rpm/pull/562

Comment 11 Fedora Update System 2018-10-03 12:35:30 UTC

python-rpm-generators-6-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-b45f3b5978

Comment 12 Fedora Update System 2018-10-03 16:31:22 UTC

python-rpm-generators-6-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-b45f3b5978

Comment 13 Fedora Update System 2018-10-07 21:00:25 UTC

python-rpm-generators-6-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.