Bug 1410631
| Summary: | pythondistdeps fails if package contains Python eggs or dist-info but doesn't BuildRequire python3-setuptools | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Adam Williamson <awilliam> |
| Component: | rpm | Assignee: | Packaging Maintenance Team <packaging-team-maint> |
| Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rawhide | CC: | cstratak, dennis, ignatenko, kardos.lubos, ngompa13, packaging-team-maint, pknirsch, pmatilai, python-sig, sgallagh, torsava |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | rpm-4.13.0-10.fc26 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-01-06 14:29:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
For the record, apparently the generator also exists for F25, but uses Python 2 there (so at least potentially, the opposite problem could occur - you could have a Python 3-only package where python2-setuptools isn't in the build environment, and get the same result). Hmm, that's interesting. I thought that pkg_resources should be part of distutils, so it would be part of system-python So, after discussing this issue on #fedora-python and #fedora-releng, we've identified several possible solutions: - Make a PEP to move setuptools into the Python standard library (good long term solution, but not feasible in the time before the mass rebuild) - Make python-setuptools depend on system-python (infeasible as it depends on unittest and lib2to3) - reimplement subset of pkg_resources inside RPM (problematic, would need to be maintained long-term) - Make rpm-build depend on python3-setuptools The last option appeared the cleanest, and thus was discussed with Dennis Gilmore who initially objected to putting the full Python stack back into the buildroot (as it was unannouncedly removed by rpm switching to system-python, which caused this problem in the first place [0]). In the end this solution was approved and put in place. [0] https://pagure.io/fesco/issue/1660 Well, having rpm-build depend on python3-setuptools is the easiest and fastest path no doubt, but it's also quite hysterical in that rpm-build itself obviously does not need python* anything. The *clean* solution would be following suite with perl and moving the python dependency generators into python and python3. (In reply to Panu Matilainen from comment #4) > Well, having rpm-build depend on python3-setuptools is the easiest and > fastest path no doubt, but it's also quite hysterical in that rpm-build > itself obviously does not need python* anything. > > The *clean* solution would be following suite with perl and moving the > python dependency generators into python and python3. That sounds intriguing, could I read more about how that is done? Or could you expand on it a bit here? Well, basically grab a copy of current python* generator rules + scripts from rpm, stick them to a separate package which rpm-build is initially made to require. Once that is established as working, cutting the ties will be a simple matter of dropping the dependency to python-generators (or whatever the name). See bug 1110823 and https://fedoraproject.org/wiki/Changes/Build_Root_Without_Perl for the perl counterpart. Oh, and of course there's the option of just splitting to python generators into a separate sub-package of rpm. That allows moving just the dependency to python, instead of moving the entire maintainership (as was done for perl) (In reply to Panu Matilainen from comment #7) > Oh, and of course there's the option of just splitting to python generators > into a separate sub-package of rpm. That allows moving just the dependency > to python, instead of moving the entire maintainership (as was done for perl) I'd rather have generators be subpackages of RPM rather than be moved out elsewhere. That way, there still remains only one place to update it. (In reply to Neal Gompa from comment #8) > (In reply to Panu Matilainen from comment #7) > > Oh, and of course there's the option of just splitting to python generators > > into a separate sub-package of rpm. That allows moving just the dependency > > to python, instead of moving the entire maintainership (as was done for perl) > > I'd rather have generators be subpackages of RPM rather than be moved out > elsewhere. That way, there still remains only one place to update it. I agree. Is there a BZ to track the splitting of the python generators open somewhere? This issue is blocking the Base Runtime effort. (In reply to Stephen Gallagher from comment #10) > Is there a BZ to track the splitting of the python generators open > somewhere? This issue is blocking the Base Runtime effort. Not to my knowledge. Just for the record wrt my earlier comments, having python depend on an rpm subcomponent would be quite problematic, at least for bootstrapping purposes whenever that is needed. I'd much rather see the generators maintained in a separate package so the python SIG can do whatever they please with them, just like perl. The number of places to update doesn't change because you dont need to update rpm anymore then. Update: The Python generators were moved to a standalone package [0] and are no longer part of the rpm-build subpackage, which as a rosult no longer requires Python. [0] https://admin.fedoraproject.org/pkgdb/package/rpms/python-rpm-generators/ How do we make sure those are available for builds that need them? Should it happen automatically somehow? (In reply to Adam Williamson from comment #13) > How do we make sure those are available for builds that need them? Should it > happen automatically somehow? The python rpm generators package is now a dependency of python-devel and python3-devel, so the packages that require it (or at least those compliant with the guidelines), shouldn't be affected. |
The newish pythondistdeps.py in rpm-build does this: if lower.endswith('.egg') or \ lower.endswith('.egg-info') or \ lower.endswith('.dist-info'): # This import is very slow, so only do it if needed from pkg_resources import Distribution, FileMetadata, PathMetadata pythondistdeps.py is run through /usr/libexec/system-python , which is python3. The python3 pkg_resources module is in python3-setuptools , which is not pulled into the default RPM build environment, via rpm-build package deps or anything else. So if you have a Python-y package which doesn't BR python3-setuptools - e.g. a Python 2-only package, like certbot - you get this spew at the end of your package build: + LICENSEDIR=/builddir/build/BUILDROOT/certbot-0.9.3-2.fc26.x86_64/usr/share/licenses/python2-certbot + export LICENSEDIR + /usr/bin/mkdir -p /builddir/build/BUILDROOT/certbot-0.9.3-2.fc26.x86_64/usr/share/licenses/python2-certbot + cp -pr LICENSE.txt /builddir/build/BUILDROOT/certbot-0.9.3-2.fc26.x86_64/usr/share/licenses/python2-certbot + exit 0 Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Traceback (most recent call last): File "/usr/lib/rpm/pythondistdeps.py", line 101, in <module> from pkg_resources import Distribution, FileMetadata, PathMetadata ModuleNotFoundError: No module named 'pkg_resources' Provides: python-certbot = 0.9.3-2.fc26 python-letsencrypt = 0.9.3-2.fc26 python2-certbot = 0.9.3-2.fc26 python2-letsencrypt = 0.9.3-2.fc26 and the deps that pythondistdeps should generate don't in fact get generated. Neal suggested just adding a Requires: python3-setuptools to rpm-build, but Dennis was very against that.