Bug 1701286

Summary: Use RPM built wheels of setuptools and pip in Python's ensurepip
Product: Red Hat Enterprise Linux 8 Reporter: Miro Hrončok <mhroncok>
Component: python3Assignee: Charalampos Stratakis <cstratak>
Status: CLOSED ERRATA QA Contact: Lukáš Zachar <lzachar>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.1CC: cstratak, hhorak, mhroncok, pviktori, torsava, yselkowi
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: python3-3.6.8-12.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-05 22:03:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1718031, 1718032, 1718033    
Bug Blocks: 1727309    

Description Miro Hrončok 2019-04-18 14:50:05 UTC
Upstream CPython bundles pre-built wheels of setuptools and pip. It installs them to the system when the ensurepip module is invoked, or to new virtual environments created via the venv module (the venv module transitively invokes the ensurepip module).

In Fedora, we remove those bundled wheels and we have a workaround.

In Fedora 28 (and earlier), we had a rather large custom patch in python3 (named "rewheel") that made the ensurepip module (and transitively venv) collect the files from the installed python3-setuptools and python3-pip packages, "rewheeled" them into an on demand wheel in /tmp and used that wheel instead of the originally bundled one.

Since Fedora 29, we do it differently. During the build of python-setuptools and python-pip RPMs a wheel is built from sources. The components gained a new subpackage python-setuptools-wheel / python-pip-wheel with the wheel:

$ rpm -ql python-pip-wheel

Python is then patched simply to look for the wheels in different location and pick the highest available version instead of the hardcoded one.

The change can be seen in https://src.fedoraproject.org/rpms/python3/pull-request/49

It has several benefits (with footnotes):

 - the patch is trivial and doesn't need much maintenance [1], whereas this was not true for the previous "rewheel" patch
 - RPM built wheels can be used in multiple Python interpreters assuming they are compatible - in Fedora they are currently compatible with Python 2.7, 3.4-3.8, both PyPys and somehow also with Jython [2]
 - when python-wheel also gains a wheel package (python-wheel-wheel), together those RPM built wheels can also be used in the python-virtualenv package - previously it just bundled the wheels from upstream
 - the mechanism in ensurepip is as much upstream-similar as possible
 - when pip or setuptools is updated after python3 is built, the version reported by the ensurepip module remains current, whereas previously it was baked in on build time
 - python3 doesn't depend on python3-pip and  python3-setuptools packages, which might be appreciated by some system administrators [3]

To lower the maintenance burden, I suppose we should update python in RHEL 8.1 to be aligned with Fedora 29+ in this manner.

Python 2 in RHEL 6/7 doesn't have pip.

Python 3 in RHEL 7 has rewheel. Whether it should stay that way or not is out of scope here.


[1] The patch needs to be rebased when new setuptools/pip releases are bundled within a new python release, however the rebasing only consists of hardcoded version bumps in the original - that serves as a reminder to also update the versions hardcoded in the specfile (used when this entire patch is bconded out)

[2] They are not compatible with Python 2.6 but that doesn't have neithr ensurepip nor venv

[3] Technically it still depends on pip and setuptools wheels, however those are not "installed" (from Python perspective) but only "available"

Comment 1 Tomas Orsava 2019-04-23 14:26:12 UTC
I propose that we do a slight adjustment for RHEL in regards to the wheel packages. As the python2 / python3 / (future) python38 / ... versions in RHEL8 should be independent, an update to setuptools/pip/wheel of one of them should not affect the others.
Therefore I believe we should instead version the packages (e.g. python3-setuptools-wheel), and make one for each Python stack in RHEL8.

This will also most likely necessitate a different file location, as `/usr/share/python-wheels/pip-19.0.3-py2.py3-none-any.whl` is not versioned either, and the files could clash.

Comment 2 Miro Hrončok 2019-04-23 14:36:12 UTC
The modules make this indeed more complicated.

Comment 3 Miro Hrončok 2019-05-07 13:20:22 UTC
I propose this path for modules:




Comment 4 Tomas Orsava 2019-05-14 21:49:28 UTC
Sounds good.

Comment 5 Petr Viktorin 2019-05-16 12:40:44 UTC
Sounds good.

Comment 7 Yaakov Selkowitz 2019-07-05 13:50:32 UTC
This change has caused a noticeable increase in the size of the ubi8 base image.  While I understand the reason for leaving Requires: platform-python-setuptools for platform-python, as many packages do rely on it at runtime, what would require pip at runtime that it too needs to be pulled in "just in case"?  

FWIW, as a point of reference in Fedora:

$ dnf repoquery --whatrequires python3-pip --releasever=30

I don't think any of those are in RHEL, and if any are, it's certainly few enough to fix individually.  (For comparison, python3-setuptools shows 384 reverse dependencies for F30.)

If the Requires: platform-python-pip (which is by far the larger of the two) would be dropped from platform-python, that would help significantly with base image size.

Comment 12 Lukáš Zachar 2019-09-27 06:17:37 UTC
I've created bug 1756217 to discuss & fix size of platform-python dependencies.

Comment 18 errata-xmlrpc 2019-11-05 22:03:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.