Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1410631 - pythondistdeps fails if package contains Python eggs or dist-info but doesn't BuildRequire python3-setuptools
Summary: pythondistdeps fails if package contains Python eggs or dist-info but doesn't...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: rawhide
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-05 23:58 UTC by Adam Williamson
Modified: 2017-05-24 16:49 UTC (History)
11 users (show)

Fixed In Version: rpm-4.13.0-10.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-06 14:29:44 UTC
Type: Bug


Attachments (Terms of Use)

Description Adam Williamson 2017-01-05 23:58:49 UTC
The newish pythondistdeps.py in rpm-build does this:

    if lower.endswith('.egg') or \
            lower.endswith('.egg-info') or \
            lower.endswith('.dist-info'):
        # This import is very slow, so only do it if needed
        from pkg_resources import Distribution, FileMetadata, PathMetadata

pythondistdeps.py is run through /usr/libexec/system-python , which is python3. The python3 pkg_resources module is in python3-setuptools , which is not pulled into the default RPM build environment, via rpm-build package deps or anything else.

So if you have a Python-y package which doesn't BR python3-setuptools - e.g. a Python 2-only package, like certbot - you get this spew at the end of your package build:

+ LICENSEDIR=/builddir/build/BUILDROOT/certbot-0.9.3-2.fc26.x86_64/usr/share/licenses/python2-certbot
+ export LICENSEDIR
+ /usr/bin/mkdir -p /builddir/build/BUILDROOT/certbot-0.9.3-2.fc26.x86_64/usr/share/licenses/python2-certbot
+ cp -pr LICENSE.txt /builddir/build/BUILDROOT/certbot-0.9.3-2.fc26.x86_64/usr/share/licenses/python2-certbot
+ exit 0
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module>
    from pkg_resources import Distribution, FileMetadata, PathMetadata
ModuleNotFoundError: No module named 'pkg_resources'
Provides: python-certbot = 0.9.3-2.fc26 python-letsencrypt = 0.9.3-2.fc26 python2-certbot = 0.9.3-2.fc26 python2-letsencrypt = 0.9.3-2.fc26

and the deps that pythondistdeps should generate don't in fact get generated.

Neal suggested just adding a Requires: python3-setuptools to rpm-build, but Dennis was very against that.

Comment 1 Adam Williamson 2017-01-06 00:22:47 UTC
For the record, apparently the generator also exists for F25, but uses Python 2 there (so at least potentially, the opposite problem could occur - you could have a Python 3-only package where python2-setuptools isn't in the build environment, and get the same result).

Comment 2 Igor Gnatenko 2017-01-06 09:31:22 UTC
Hmm, that's interesting.

I thought that pkg_resources should be part of distutils, so it would be part of system-python

Comment 3 Tomas Orsava 2017-01-06 14:29:44 UTC
So, after discussing this issue on #fedora-python and #fedora-releng, we've identified several possible solutions:

- Make a PEP to move setuptools into the Python standard library (good long term solution, but not feasible in the time before the mass rebuild)
- Make python-setuptools depend on system-python (infeasible as it depends on unittest and lib2to3)
- reimplement subset of pkg_resources inside RPM (problematic, would need to be maintained long-term)
- Make rpm-build depend on python3-setuptools

The last option appeared the cleanest, and thus was discussed with Dennis Gilmore who initially objected to putting the full Python stack back into the buildroot (as it was unannouncedly removed by rpm switching to system-python, which caused this problem in the first place [0]). In the end this solution was approved and put in place.

[0] https://pagure.io/fesco/issue/1660

Comment 4 Panu Matilainen 2017-01-09 07:08:13 UTC
Well, having rpm-build depend on python3-setuptools is the easiest and fastest path no doubt, but it's also quite hysterical in that rpm-build itself obviously does not need python* anything.

The *clean* solution would be following suite with perl and moving the python dependency generators into python and python3.

Comment 5 Tomas Orsava 2017-01-09 09:34:58 UTC
(In reply to Panu Matilainen from comment #4)
> Well, having rpm-build depend on python3-setuptools is the easiest and
> fastest path no doubt, but it's also quite hysterical in that rpm-build
> itself obviously does not need python* anything.
> 
> The *clean* solution would be following suite with perl and moving the
> python dependency generators into python and python3.

That sounds intriguing, could I read more about how that is done? Or could you expand on it a bit here?

Comment 6 Panu Matilainen 2017-01-09 09:41:01 UTC
Well, basically grab a copy of current python* generator rules + scripts from rpm, stick them to a separate package which rpm-build is initially made to require. Once that is established as working, cutting the ties will be a simple matter of dropping the dependency to python-generators (or whatever the name).

See bug 1110823 and https://fedoraproject.org/wiki/Changes/Build_Root_Without_Perl for the perl counterpart.

Comment 7 Panu Matilainen 2017-01-09 09:43:52 UTC
Oh, and of course there's the option of just splitting to python generators into a separate sub-package of rpm. That allows moving just the dependency to python, instead of moving the entire maintainership (as was done for perl)

Comment 8 Neal Gompa 2017-01-09 12:23:28 UTC
(In reply to Panu Matilainen from comment #7)
> Oh, and of course there's the option of just splitting to python generators
> into a separate sub-package of rpm. That allows moving just the dependency
> to python, instead of moving the entire maintainership (as was done for perl)

I'd rather have generators be subpackages of RPM rather than be moved out elsewhere. That way, there still remains only one place to update it.

Comment 9 Tomas Orsava 2017-01-10 13:46:04 UTC
(In reply to Neal Gompa from comment #8)
> (In reply to Panu Matilainen from comment #7)
> > Oh, and of course there's the option of just splitting to python generators
> > into a separate sub-package of rpm. That allows moving just the dependency
> > to python, instead of moving the entire maintainership (as was done for perl)
> 
> I'd rather have generators be subpackages of RPM rather than be moved out
> elsewhere. That way, there still remains only one place to update it.

I agree.

Comment 10 Stephen Gallagher 2017-03-06 20:14:25 UTC
Is there a BZ to track the splitting of the python generators open somewhere? This issue is blocking the Base Runtime effort.

Comment 11 Panu Matilainen 2017-03-07 07:44:50 UTC
(In reply to Stephen Gallagher from comment #10)
> Is there a BZ to track the splitting of the python generators open
> somewhere? This issue is blocking the Base Runtime effort.

Not to my knowledge.

Just for the record wrt my earlier comments, having python depend on an rpm subcomponent would be quite problematic, at least for bootstrapping purposes whenever that is needed.

I'd much rather see the generators maintained in a separate package so the python SIG can do whatever they please with them, just like perl. The number of places to update doesn't change because you dont need to update rpm anymore then.

Comment 12 Tomas Orsava 2017-05-24 14:29:00 UTC
Update:
The Python generators were moved to a standalone package [0] and are no longer part of the rpm-build subpackage, which as a rosult no longer requires Python.

[0] https://admin.fedoraproject.org/pkgdb/package/rpms/python-rpm-generators/

Comment 13 Adam Williamson 2017-05-24 16:18:56 UTC
How do we make sure those are available for builds that need them? Should it happen automatically somehow?

Comment 14 Charalampos Stratakis 2017-05-24 16:49:16 UTC
(In reply to Adam Williamson from comment #13)
> How do we make sure those are available for builds that need them? Should it
> happen automatically somehow?

The python rpm generators package is now a dependency of python-devel and python3-devel, so the packages that require it (or at least those compliant with the guidelines), shouldn't be affected.


Note You need to log in before you can comment on or make changes to this bug.