Bug 1464570 - ensurepip python3 package missing _bundled directory
ensurepip python3 package missing _bundled directory
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: python3 (Show other bugs)
26
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Charalampos Stratakis
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-23 15:25 EDT by Antoine Cotten
Modified: 2018-01-24 08:18 EST (History)
12 users (show)

See Also:
Fixed In Version: python3-3.6.1-7.fc26
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-01-23 17:06:28 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Python 29523 None None None 2017-06-23 16:30 EDT

  None (edit)
Description Antoine Cotten 2017-06-23 15:25:11 EDT
Description of problem:

The python3 'ensurepip' package shipped as part of python3-libs is missing the '_bundled' directory.

cf. https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled

This breaks built-in modules like 'venv'.


Version-Release number of selected component (if applicable):

3.6.1-6.fc26.x86_64


How reproducible:

-- Try venv creation -----------------------------

~$ python3 -m venv py36

Error: Command '['/home/acotten/py36/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.


-- Manually execute the above command ------------

~$ py36/bin/python3 -Im ensurepip

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 224, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 121, in bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-34.2.0-py2.py3-none-any.whl'


-- List package directory ------------------------

~$ ls -1 /usr/lib64/python3.6/ensurepip/

__pycache__
rewheel
__init__.py
__main__.py
_uninstall.py


Actual results:

The '/usr/lib64/python3.6/ensurepip/_bundle' directory is missing.


Expected results:

The '/usr/lib64/python3.6/ensurepip/_bundle' directory ships as part of the python3-libs package.
Comment 1 Antoine Cotten 2017-06-23 16:09:49 EDT
This whole thing looks really flawed to me.

1. 'ensurepip' is called with the following args from 'venv':
https://github.com/python/cpython/blob/3.6/Lib/venv/__init__.py#L243-L244

2. The expected bundled versions are hardcoded in 'ensurepip':
https://github.com/python/cpython/blob/3.6/Lib/ensurepip/__init__.py#L11-L18

3. 'ensurepip' always tries to open the bundled wheels:
https://github.com/python/cpython/blob/3.6/Lib/ensurepip/__init__.py#L80-L87

4. Fedora requires different bundled versions than upstream:
http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00260-require-setuptools-dependencies.patch?h=f26#n9

This means we must pull the 'setuptools', 'pip', etc. into the _bundled directory during the package build.
Comment 2 Charalampos Stratakis 2017-06-24 11:16:27 EDT
This is an issue with setuptools and its dependencies (upstream unbundled them and then rebundled them again) [0] [1].

Already have a fix for it, however I was blocked on creating a new build due to http://bugs.python.org/issue30714 which has been fixed now. Will push a new build soon.

Our python3 version doesn't use the bundled wheels of setuptools and pip (And that's why we do not ship the _bundled directory [2]), instead it uses the system installed wheels, which are then installed inside the venvs (rewheel patch [3]).

[0] http://pkgs.fedoraproject.org/cgit/rpms/python-setuptools.git/diff/python-setuptools.spec?id=49b1915e5e251e00a6fc95e78c60e611f633e2cf

[1] http://pkgs.fedoraproject.org/cgit/rpms/python-setuptools.git/commit/?id=5e5ba5a5931b254843baaf45e6dbd1ebe2769b91

[2] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec#n1304

[3] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00189-add-rewheel-module.patch
Comment 3 Fedora Update System 2017-06-25 06:16:05 EDT
python3-3.6.1-7.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-cffdee8d35
Comment 4 Fedora Update System 2017-06-25 21:22:13 EDT
python3-3.6.1-7.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-cffdee8d35
Comment 5 Fedora Update System 2017-06-26 15:13:30 EDT
python3-3.6.1-7.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 6 Jon Dufresne 2017-07-08 12:59:41 EDT
I'm still experiencing this issue with package python3-3.6.1-8.fc26.x86_64

Observe:

$ python3 -m venv venv
Error: Command '['/home/jon/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
Comment 7 Antoine Cotten 2017-07-08 13:04:42 EDT
I can't reproduce with the version you mentioned.

What does "/home/jon/venv/bin/python3 -Im ensurepip --upgrade --default-pip" return?
Comment 8 Jon Dufresne 2017-07-08 13:10:20 EDT
$ /home/jon/venv/bin/python3 -Im ensurepip --upgrade --default-pip
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 212, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 109, in bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-28.8.0-py2.py3-none-any.whl'


I installed Fedora 26 from scratch (not an upgrade) yesterday.
Comment 9 Jon Dufresne 2017-07-09 13:34:17 EDT
Should I reopen this as I can still reproduce? Sorry, I'm not sure what correct procedure is.
Comment 10 Charalampos Stratakis 2017-07-10 08:11:09 EDT
(In reply to Jon Dufresne from comment #9)
> Should I reopen this as I can still reproduce? Sorry, I'm not sure what
> correct procedure is.

Hello.

I can't reproduce it as well. Is this happening on a fresh F26 install? Did you update pip or setuptools somehow through pip?
Comment 11 Jon Dufresne 2017-07-10 10:10:08 EDT
> Is this happening on a fresh F26 install?

Yes. It was installed from scratch on July 8.

> Did you update pip or setuptools somehow through pip?

No. If there is some way I could verify that my system isn't unintentionally modified, perhaps through check sums, please let me know.
Comment 12 Miro Hrončok 2017-07-10 10:51:16 EDT
Try:

$ rpm -V $(rpm -qa | grep python3)
Comment 13 Jon Dufresne 2017-07-11 22:12:02 EDT
I reinstalled Fedora 26 from scratch today with the official release. I can no longer reproduce. Not sure what I was experiencing, but can't reproduce it anymore. Thanks for your time and help.
Comment 14 Stefan Midjich 2018-01-23 13:03:08 EST
Found this on google and was surprised of the mixed results because I've been able to reproduce this for as long as I can remember using F26. At least.

$ python3 -V
Python 3.6.3
$ python3 -m venv .venv
Error: Command '['/home/stemid/Utveckling/test/.venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
$ source .venv/bin/activate
-bash: .venv/bin/activate: No such file or directory

Here is my workaround that I found online.

$ python3 -m venv --without-pip .venv
$ source .venv/bin/activate
$ curl https://bootstrap.pypa.io/get-pip.py | python
Comment 15 Stefan Midjich 2018-01-23 13:04:07 EST
Forgot to mention:

python3.x86_64                         3.6.3-2.fc26

Fedora 26 (Workstation Edition)
Comment 16 Miro Hrončok 2018-01-23 14:54:46 EST
Try:

$ rpm -V $(rpm -qa | grep python3)
Comment 17 Stefan Midjich 2018-01-23 15:27:49 EST
(In reply to Miro Hrončok from comment #16)
> Try:
> 
> $ rpm -V $(rpm -qa | grep python3)

$ rpm -V $(rpm -qa | grep python3)
S.5....T.    /usr/lib/python3.6/site-packages/urllib3/packages/__pycache__/six.cpython-36.pyc

What does that mean? A leftover compiled file that should be removed?
Comment 18 Miro Hrončok 2018-01-23 15:43:41 EST
One pyc shouldn't be a problem.

How about:

LANG=C.utf8 find /usr/lib64/python3.6/site-packages/ -exec rpm -qf {} \; | grep "is not owned by any package"

and:

LANG=C.utf8 find /usr/lib/python3.6/site-packages/ -exec rpm -qf {} \; | grep "is not owned by any package"
Comment 19 Stefan Midjich 2018-01-23 16:22:00 EST
$ LANG=C.utf8 find /usr/lib64/python3.6/site-packages/ -exec rpm -qf {} \; | grep "is not owned by any package"
file /usr/lib64/python3.6/site-packages/report is not owned by any package
file /usr/lib64/python3.6/site-packages/reportclient is not owned by any package

$ LANG=C.utf8 find /usr/lib/python3.6/site-packages/ -exec rpm -qf {} \; | grep "is not owned by any package"
file /usr/lib/python3.6/site-packages/isc is not owned by any package
file /usr/lib/python3.6/site-packages/isc/__pycache__ is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale/ru_RU is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale/ru_RU/LC_MESSAGES is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale/ko_KR is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale/ko_KR/LC_MESSAGES is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale/fr_FR is not owned by any package
file /usr/lib/python3.6/site-packages/humanize/locale/fr_FR/LC_MESSAGES is not owned by any package
file /usr/lib/python3.6/site-packages/pykickstart is not owned by any package
file /usr/lib/python3.6/site-packages/pykickstart/handlers is not owned by any package
file /usr/lib/python3.6/site-packages/pykickstart/commands is not owned by any package
file /usr/lib/python3.6/site-packages/sepolicy is not owned by any package
file /usr/lib/python3.6/site-packages/nbconvert is not owned by any package

If it's noteworthy, I believe I've upgraded this Fedora since maybe 24 or 23 without a clean re-install.

But those leftover python packages seem to be no older than 2017. One had a couple of .py files from 2015 but that was the only exception.
Comment 20 Miro Hrončok 2018-01-23 16:43:40 EST
None of those seems dangerous. Confusing.

What is the exact error if you do:

$ python3 -Im ensurpip
Comment 21 Stefan Midjich 2018-01-23 17:01:10 EST
(In reply to Miro Hrončok from comment #20)
> None of those seems dangerous. Confusing.
> 
> What is the exact error if you do:
> 
> $ python3 -Im ensurpip

I assume you meant outside of virtualenv.

$ python3 -Im ensurpip
/usr/bin/python3: No module named ensurpip

Inside virtualenv is the same except the python path is different. This is the virtualenv where I've used curl to workaround the issue.
Comment 22 Stefan Midjich 2018-01-23 17:02:18 EST
(In reply to Miro Hrončok from comment #20)
> None of those seems dangerous. Confusing.
> 
> What is the exact error if you do:
> 
> $ python3 -Im ensurpip

Eh typo;

$ python3 -Im ensurepip       
Traceback (most recent call last):                                    
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main                                                                     
    "__main__", mod_spec)                                             
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code         
    exec(code, run_globals)                                           
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 4, in <module>                                                                     
    ensurepip._main()                                                 
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 212, in _main                                                                      
    default_pip=args.default_pip,                                     
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 91, in bootstrap                                                                   
    new_whl = rewheel.rewheel_from_record(dr, rewheel_dir.name)       
  File "/usr/lib64/python3.6/ensurepip/rewheel/__init__.py", line 75, in rewheel_from_record                                                 
    new_wheel.write(os.path.join(site_dir, f), arcname=f)             
  File "/usr/lib64/python3.6/zipfile.py", line 1594, in write         
    zinfo = ZipInfo.from_file(filename, arcname)                      
  File "/usr/lib64/python3.6/zipfile.py", line 484, in from_file      
    st = os.stat(filename)                                            
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3.6/site-packages/pip/_vendor/ipaddress.py'
Comment 23 Miro Hrončok 2018-01-23 17:06:28 EST
Ha!

That was fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2017-eac4d90de1 https://bugzilla.redhat.com/show_bug.cgi?id=1522684

Just dnf update python3.
Comment 24 Stefan Midjich 2018-01-23 17:09:58 EST
(In reply to Miro Hrončok from comment #23)
> Ha!
> 
> That was fixed in
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-eac4d90de1
> https://bugzilla.redhat.com/show_bug.cgi?id=1522684
> 
> Just dnf update python3.

There are no updates for python3 in dnf. 

I'm on F26. 

$ LANG=C sudo dnf update python3
Last metadata expiration check: 0:10:48 ago on Tue Jan 23 22:59:06 2018.
Dependencies resolved.
Nothing to do.
Complete!
Comment 25 Miro Hrončok 2018-01-23 17:14:59 EST
oh sorry, sudo dnf update python3-pip
Comment 26 Stefan Midjich 2018-01-24 08:18:03 EST
(In reply to Stefan Midjich from comment #24)
> (In reply to Miro Hrončok from comment #23)
> > Ha!
> > 
> > That was fixed in
> > https://bodhi.fedoraproject.org/updates/FEDORA-2017-eac4d90de1
> > https://bugzilla.redhat.com/show_bug.cgi?id=1522684
> > 
> > Just dnf update python3.
> 
> There are no updates for python3 in dnf. 
> 
> I'm on F26. 
> 
> $ LANG=C sudo dnf update python3
> Last metadata expiration check: 0:10:48 ago on Tue Jan 23 22:59:06 2018.
> Dependencies resolved.
> Nothing to do.
> Complete!

That worked better, now it's resolved. Thanks. :)

Note You need to log in before you can comment on or make changes to this bug.