Bug 1464570 - ensurepip python3 package missing _bundled directory
ensurepip python3 package missing _bundled directory
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: python3 (Show other bugs)
26
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Charalampos Stratakis
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-23 15:25 EDT by Antoine Cotten
Modified: 2017-07-11 22:12 EDT (History)
11 users (show)

See Also:
Fixed In Version: python3-3.6.1-7.fc26
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-26 15:13:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Python 29523 None None None 2017-06-23 16:30 EDT

  None (edit)
Description Antoine Cotten 2017-06-23 15:25:11 EDT
Description of problem:

The python3 'ensurepip' package shipped as part of python3-libs is missing the '_bundled' directory.

cf. https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled

This breaks built-in modules like 'venv'.


Version-Release number of selected component (if applicable):

3.6.1-6.fc26.x86_64


How reproducible:

-- Try venv creation -----------------------------

~$ python3 -m venv py36

Error: Command '['/home/acotten/py36/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.


-- Manually execute the above command ------------

~$ py36/bin/python3 -Im ensurepip

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 224, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 121, in bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-34.2.0-py2.py3-none-any.whl'


-- List package directory ------------------------

~$ ls -1 /usr/lib64/python3.6/ensurepip/

__pycache__
rewheel
__init__.py
__main__.py
_uninstall.py


Actual results:

The '/usr/lib64/python3.6/ensurepip/_bundle' directory is missing.


Expected results:

The '/usr/lib64/python3.6/ensurepip/_bundle' directory ships as part of the python3-libs package.
Comment 1 Antoine Cotten 2017-06-23 16:09:49 EDT
This whole thing looks really flawed to me.

1. 'ensurepip' is called with the following args from 'venv':
https://github.com/python/cpython/blob/3.6/Lib/venv/__init__.py#L243-L244

2. The expected bundled versions are hardcoded in 'ensurepip':
https://github.com/python/cpython/blob/3.6/Lib/ensurepip/__init__.py#L11-L18

3. 'ensurepip' always tries to open the bundled wheels:
https://github.com/python/cpython/blob/3.6/Lib/ensurepip/__init__.py#L80-L87

4. Fedora requires different bundled versions than upstream:
http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00260-require-setuptools-dependencies.patch?h=f26#n9

This means we must pull the 'setuptools', 'pip', etc. into the _bundled directory during the package build.
Comment 2 Charalampos Stratakis 2017-06-24 11:16:27 EDT
This is an issue with setuptools and its dependencies (upstream unbundled them and then rebundled them again) [0] [1].

Already have a fix for it, however I was blocked on creating a new build due to http://bugs.python.org/issue30714 which has been fixed now. Will push a new build soon.

Our python3 version doesn't use the bundled wheels of setuptools and pip (And that's why we do not ship the _bundled directory [2]), instead it uses the system installed wheels, which are then installed inside the venvs (rewheel patch [3]).

[0] http://pkgs.fedoraproject.org/cgit/rpms/python-setuptools.git/diff/python-setuptools.spec?id=49b1915e5e251e00a6fc95e78c60e611f633e2cf

[1] http://pkgs.fedoraproject.org/cgit/rpms/python-setuptools.git/commit/?id=5e5ba5a5931b254843baaf45e6dbd1ebe2769b91

[2] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec#n1304

[3] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00189-add-rewheel-module.patch
Comment 3 Fedora Update System 2017-06-25 06:16:05 EDT
python3-3.6.1-7.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-cffdee8d35
Comment 4 Fedora Update System 2017-06-25 21:22:13 EDT
python3-3.6.1-7.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-cffdee8d35
Comment 5 Fedora Update System 2017-06-26 15:13:30 EDT
python3-3.6.1-7.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 6 Jon Dufresne 2017-07-08 12:59:41 EDT
I'm still experiencing this issue with package python3-3.6.1-8.fc26.x86_64

Observe:

$ python3 -m venv venv
Error: Command '['/home/jon/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
Comment 7 Antoine Cotten 2017-07-08 13:04:42 EDT
I can't reproduce with the version you mentioned.

What does "/home/jon/venv/bin/python3 -Im ensurepip --upgrade --default-pip" return?
Comment 8 Jon Dufresne 2017-07-08 13:10:20 EDT
$ /home/jon/venv/bin/python3 -Im ensurepip --upgrade --default-pip
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 212, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 109, in bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-28.8.0-py2.py3-none-any.whl'


I installed Fedora 26 from scratch (not an upgrade) yesterday.
Comment 9 Jon Dufresne 2017-07-09 13:34:17 EDT
Should I reopen this as I can still reproduce? Sorry, I'm not sure what correct procedure is.
Comment 10 Charalampos Stratakis 2017-07-10 08:11:09 EDT
(In reply to Jon Dufresne from comment #9)
> Should I reopen this as I can still reproduce? Sorry, I'm not sure what
> correct procedure is.

Hello.

I can't reproduce it as well. Is this happening on a fresh F26 install? Did you update pip or setuptools somehow through pip?
Comment 11 Jon Dufresne 2017-07-10 10:10:08 EDT
> Is this happening on a fresh F26 install?

Yes. It was installed from scratch on July 8.

> Did you update pip or setuptools somehow through pip?

No. If there is some way I could verify that my system isn't unintentionally modified, perhaps through check sums, please let me know.
Comment 12 Miro Hrončok 2017-07-10 10:51:16 EDT
Try:

$ rpm -V $(rpm -qa | grep python3)
Comment 13 Jon Dufresne 2017-07-11 22:12:02 EDT
I reinstalled Fedora 26 from scratch today with the official release. I can no longer reproduce. Not sure what I was experiencing, but can't reproduce it anymore. Thanks for your time and help.

Note You need to log in before you can comment on or make changes to this bug.