Bug 1701286 - Use RPM built wheels of setuptools and pip in Python's ensurepip
Summary: Use RPM built wheels of setuptools and pip in Python's ensurepip
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: python3
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Charalampos Stratakis
QA Contact: Lukáš Zachar
URL:
Whiteboard:
Depends On: 1718031 1718032 1718033
Blocks: 1727309
TreeView+ depends on / blocked
 
Reported: 2019-04-18 14:50 UTC by Miro Hrončok
Modified: 2019-11-05 22:03 UTC (History)
6 users (show)

Fixed In Version: python3-3.6.8-12.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-05 22:03:43 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:3520 None None None 2019-11-05 22:03:58 UTC

Description Miro Hrončok 2019-04-18 14:50:05 UTC
Upstream CPython bundles pre-built wheels of setuptools and pip. It installs them to the system when the ensurepip module is invoked, or to new virtual environments created via the venv module (the venv module transitively invokes the ensurepip module).

In Fedora, we remove those bundled wheels and we have a workaround.

In Fedora 28 (and earlier), we had a rather large custom patch in python3 (named "rewheel") that made the ensurepip module (and transitively venv) collect the files from the installed python3-setuptools and python3-pip packages, "rewheeled" them into an on demand wheel in /tmp and used that wheel instead of the originally bundled one.

Since Fedora 29, we do it differently. During the build of python-setuptools and python-pip RPMs a wheel is built from sources. The components gained a new subpackage python-setuptools-wheel / python-pip-wheel with the wheel:

$ rpm -ql python-pip-wheel
/usr/share/licenses/python-pip-wheel
/usr/share/licenses/python-pip-wheel/LICENSE.txt
/usr/share/python-wheels
/usr/share/python-wheels/pip-19.0.3-py2.py3-none-any.whl


Python is then patched simply to look for the wheels in different location and pick the highest available version instead of the hardcoded one.

The change can be seen in https://src.fedoraproject.org/rpms/python3/pull-request/49

It has several benefits (with footnotes):

 - the patch is trivial and doesn't need much maintenance [1], whereas this was not true for the previous "rewheel" patch
 - RPM built wheels can be used in multiple Python interpreters assuming they are compatible - in Fedora they are currently compatible with Python 2.7, 3.4-3.8, both PyPys and somehow also with Jython [2]
 - when python-wheel also gains a wheel package (python-wheel-wheel), together those RPM built wheels can also be used in the python-virtualenv package - previously it just bundled the wheels from upstream
 - the mechanism in ensurepip is as much upstream-similar as possible
 - when pip or setuptools is updated after python3 is built, the version reported by the ensurepip module remains current, whereas previously it was baked in on build time
 - python3 doesn't depend on python3-pip and  python3-setuptools packages, which might be appreciated by some system administrators [3]


To lower the maintenance burden, I suppose we should update python in RHEL 8.1 to be aligned with Fedora 29+ in this manner.

Python 2 in RHEL 6/7 doesn't have pip.

Python 3 in RHEL 7 has rewheel. Whether it should stay that way or not is out of scope here.


------------------------

[1] The patch needs to be rebased when new setuptools/pip releases are bundled within a new python release, however the rebasing only consists of hardcoded version bumps in the original - that serves as a reminder to also update the versions hardcoded in the specfile (used when this entire patch is bconded out)

[2] They are not compatible with Python 2.6 but that doesn't have neithr ensurepip nor venv

[3] Technically it still depends on pip and setuptools wheels, however those are not "installed" (from Python perspective) but only "available"

Comment 1 Tomas Orsava 2019-04-23 14:26:12 UTC
I propose that we do a slight adjustment for RHEL in regards to the wheel packages. As the python2 / python3 / (future) python38 / ... versions in RHEL8 should be independent, an update to setuptools/pip/wheel of one of them should not affect the others.
Therefore I believe we should instead version the packages (e.g. python3-setuptools-wheel), and make one for each Python stack in RHEL8.

This will also most likely necessitate a different file location, as `/usr/share/python-wheels/pip-19.0.3-py2.py3-none-any.whl` is not versioned either, and the files could clash.

Comment 2 Miro Hrončok 2019-04-23 14:36:12 UTC
The modules make this indeed more complicated.

Comment 3 Miro Hrončok 2019-05-07 13:20:22 UTC
I propose this path for modules:

    /usr/share/<module_name>-wheels/

E.g.

    /usr/share/python36-wheels/pip-19.0.3-py2.py3-none-any.whl

Comment 4 Tomas Orsava 2019-05-14 21:49:28 UTC
Sounds good.

Comment 5 Petr Viktorin 2019-05-16 12:40:44 UTC
Sounds good.

Comment 7 Yaakov Selkowitz 2019-07-05 13:50:32 UTC
This change has caused a noticeable increase in the size of the ubi8 base image.  While I understand the reason for leaving Requires: platform-python-setuptools for platform-python, as many packages do rely on it at runtime, what would require pip at runtime that it too needs to be pulled in "just in case"?  

FWIW, as a point of reference in Fedora:

$ dnf repoquery --whatrequires python3-pip --releasever=30
hatch-0:0.20.0-9.fc30.noarch
mozilla-iot-gateway-addon-python-0:0.8.0-2.fc30.noarch
parsero-0:0.81-12.fc30.noarch
pipenv-0:2018.11.26-7.fc30.noarch
python3-Lektor-0:3.0.1-6.fc30.noarch
python3-django-markdownx-0:2.0.28-1.fc30.noarch
python3-pipdeptree-0:0.13.1-3.fc30.noarch
python3-pipdeptree-0:0.13.2-1.fc30.noarch
python3-pypandoc-0:1.4-7.fc30.noarch
python3-virtualenv-api-0:2.1.16-6.fc30.noarch
thonny-0:3.1.1-1.fc30.noarch

I don't think any of those are in RHEL, and if any are, it's certainly few enough to fix individually.  (For comparison, python3-setuptools shows 384 reverse dependencies for F30.)

If the Requires: platform-python-pip (which is by far the larger of the two) would be dropped from platform-python, that would help significantly with base image size.

Comment 12 Lukáš Zachar 2019-09-27 06:17:37 UTC
I've created bug 1756217 to discuss & fix size of platform-python dependencies.

Comment 18 errata-xmlrpc 2019-11-05 22:03:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3520


Note You need to log in before you can comment on or make changes to this bug.