Bug 1263057 - pyvenv3.4 doesn't work without pip
pyvenv3.4 doesn't work without pip
Status: CLOSED ERRATA
Product: Fedora EPEL
Classification: Fedora
Component: python34 (Show other bugs)
epel7
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Python Maintainers
Fedora Extras Quality Assurance
:
: 1319963 1456285 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-15 00:13 EDT by Jamie Lennox
Modified: 2017-06-02 15:18 EDT (History)
18 users (show)

See Also:
Fixed In Version: python34-3.4.5-4.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-02 15:18:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
only exclude _bundled directory when rewheel is enabled (498 bytes, patch)
2017-01-10 11:55 EST, Carl George
no flags Details | Diff

  None (edit)
Description Jamie Lennox 2015-09-15 00:13:21 EDT
Using python34 3.4.3-2.el7 on RHEL7 I tried to create a virtualenv with the provided:

pyvenv-3.4 venv3

and get the error message: 

Error: Command '['/home/cloud-user/v3/bin/python3.4', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

Which when i run myself i get: 

Traceback (most recent call last):
  File "/usr/lib64/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.4/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/usr/lib64/python3.4/ensurepip/__init__.py", line 209, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.4/ensurepip/__init__.py", line 98, in bootstrap
    "_bundled/{}".format(wheel_name),
  File "/usr/lib64/python3.4/pkgutil.py", line 629, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap>", line 1623, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.4/ensurepip/_bundled/setuptools-12.0.5-py2.py3-none-any.whl'


The /usr/lib64/python3.4/ensurepip/ directory contains: 

__init__.py  __main__.py  __pycache__  _uninstall.py

There doesn't seem to be a python34?-pip or a setuptools package that would let me get passed this.
Comment 1 Carl George 2015-10-10 14:54:26 EDT
We ran into this same problem in IUS.  We fixed it by wraping this exclude with conditionals in the `%libs files` section.

    +%if 0%{?with_rewheel}
     %exclude %{pylibdir}/ensurepip/_bundled
    +%else
    +%{pylibdir}/ensurepip/_bundled
    +%endif
Comment 3 hleeney 2016-01-08 06:39:14 EST
Hey Carl, can you elaborate a little as to where you mad that change?
Comment 4 hleeney 2016-01-08 06:39:37 EST
Hey Carl, can you elaborate a little as to where you made that change?
Comment 5 Carl George 2016-01-08 09:56:07 EST
Howdy Hugo,

Sure thing.

https://github.com/iuscommunity-pkg/python34u/commit/955cd1a7c63211994bcf608f463b6661282c88e2
https://github.com/iuscommunity-pkg/python34u/commit/48f3316eeeb0a4ccc37fe630e10ebaa2c7d7ca3e

I should of just included those in my originally comment, sorry about that.
Comment 7 Felix Schwarz 2017-01-10 06:01:27 EST
I think EPEL7 should at least remove /usr/bin/pyvenv in its RPMs as this is currently broken.

@python-maint: Are you looking for help with this issue? Any info on the priority regarding this? Currently it is impossible to create Python 3 virtualenvs in RHEL/CentOS 7 (partly) due this bug.
Comment 8 Felix Schwarz 2017-01-10 06:07:45 EST
Also the root cause seems to be bug 1319963 (installing python34-pip does not fix this bug).

(In reply to Carl George from comment #1)
> We ran into this same problem in IUS.  We fixed it by wraping this exclude
> with conditionals in the `%libs files` section.
> 
>     +%if 0%{?with_rewheel}
>      %exclude %{pylibdir}/ensurepip/_bundled
>     +%else
>     +%{pylibdir}/ensurepip/_bundled
>     +%endif

So basically you are just shipping the "bundled" setuptools? Personally I would prefer to not ship bundled software but I guess that is okay for Fedora these days.
Comment 9 Tomas Orsava 2017-01-10 09:06:34 EST
(In reply to Felix Schwarz from comment #7)
> I think EPEL7 should at least remove /usr/bin/pyvenv in its RPMs as this is
> currently broken.
> 
> @python-maint: Are you looking for help with this issue? Any info on the
> priority regarding this? Currently it is impossible to create Python 3
> virtualenvs in RHEL/CentOS 7 (partly) due this bug.

I'm not sure we have the capacity to tackle this issue at this time, are you interested in looking into it?
Comment 10 Carl George 2017-01-10 11:53:54 EST
Felix,

No matter how you slice it, pyvenv needs wheels of pip and setuptools to create a virtual environment.  The Python source tarball has them included ("bundled", but not in the traditional sense of the word).  Fedora has this feature called rewheel that lets pyvenv recreate wheels from the corresponding system packages, and thus they can delete the included wheels from the source tarball.  That is also present (but disabled) in python34.  Enabling rewheel requires a convoluted bootstrap process.

1. build python
2. build setuptools
3. build pip
4. build wheel
5. rebuild setuptools via bdist_wheel
6. rebuild pip via bdist_wheel
7. rebuild python with rewheel patch

EPEL's python34 is stuck at step 3.  The problem is that the spec file unconditionally excludes the wheel files.  That breaks pyvenv in the python build from step 1.  The solution is a conditional that includes the _bundled directory when not applying the rewheel patch.

+%if 0%{?with_rewheel}
 %exclude %{pylibdir}/ensurepip/_bundled
+%else
+%{pylibdir}/ensurepip/_bundled
+%endif
Comment 11 Carl George 2017-01-10 11:55 EST
Created attachment 1239160 [details]
only exclude _bundled directory when rewheel is enabled
Comment 12 Felix Schwarz 2017-01-10 12:02:49 EST
(In reply to Carl George from comment #10)
> No matter how you slice it, pyvenv needs wheels of pip and setuptools to
> create a virtual environment.  The Python source tarball has them included
> ("bundled", but not in the traditional sense of the word).  Fedora has this
> feature called rewheel that lets pyvenv recreate wheels from the
> corresponding system packages, and thus they can delete the included wheels
> from the source tarball.

I'm aware of that but the current situation is even a bit more convoluted:
- The wheel patch was included conditionally hence it is missing from Python 3 (bug 1411732 for Fedora rawhide)
- setuptools and pip rpms are not shipping dist-info (presumably because python-wheel has not python34 subpackage in EPEL7) hence even a working rewheel patch won't fix the issue (as mentioned in bug 1319963 comment 4).

Nothing spectacular just a bit of tedious work with long compile times.

Not sure what the exact bundling policy is these days. Not shipping wheel/pip twice would be very good for a medium-term solution (more to keep "this is the default version of X" than being concerned with disk space).
Comment 13 Carl George 2017-01-10 12:52:45 EST
Yup, it's messy, but at the very least the _bundled directory should be included when the rewheel patch isn't applied, so as to not break pyvenv.

> Not sure what the exact bundling policy is these days.

https://fedoraproject.org/wiki/Packaging:Guidelines#Bundling_and_Duplication_of_system_libraries

That is open to interpretation of course.  By nature a virtualenv can duplicate system libraries and is essentially user data, which is why I said this isn't traditional bundling.
Comment 14 Felix Schwarz 2017-01-10 16:58:17 EST
(In reply to Carl George from comment #13)
> Yup, it's messy, but at the very least the _bundled directory should be
> included when the rewheel patch isn't applied, so as to not break pyvenv.

agreed

> https://fedoraproject.org/wiki/Packaging:
> Guidelines#Bundling_and_Duplication_of_system_libraries
> 
> That is open to interpretation of course.  By nature a virtualenv can
> duplicate system libraries and is essentially user data, which is why I said
> this isn't traditional bundling.

I don't care so much what is in the virtualenv after it is installed but at the time of creation I think it should (preferably) use the system version. (Ideally the virtualenv had also better tooling to keep its contents up to date but that is way out of scope here.)
Comment 15 Bowe Strickland 2017-05-28 14:39:42 EDT
*** Bug 1456285 has been marked as a duplicate of this bug. ***
Comment 16 Bowe Strickland 2017-05-28 14:43:11 EDT
I'm not understanding the reluctance to include the bundled wheels.   Creating a venv implies a next step of installing packages from the network, and the first step will be to upgrade pip from the network.  Including the bundled wheels allows this bootstrapping to happen, where otherwise you're dead in the water.
Comment 17 Carl George 2017-05-29 00:01:26 EDT
Unfortunately it doesn't appear the python34 maintainers have any interest in fixing this bug, despite the fact that I've already provided the solution in the attached patch.  I'm generally not a fan of going to a project's bug tracker and recommending a "competing" solution, but I'll make an exception in this case.  If you want a working Python 3.4 venv module, just use python34u from the IUS repository.  We also have the same for Python 3.5 and 3.6 as well.

https://ius.io
Comment 18 Fedora Update System 2017-05-29 13:13:36 EDT
python34-3.4.5-4.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-1e0d75a462
Comment 19 Charalampos Stratakis 2017-05-29 13:16:35 EDT
Apologies for not getting to it sooner, this task was on my TODO list for a long time and it fell of the radar.

Pushed a fix [0], things should work properly now.

[0] http://pkgs.fedoraproject.org/cgit/rpms/python34.git/commit/?h=epel7&id=4596d5c7ca35c30910fcc85bc8f16a0bc0715661
Comment 20 Charalampos Stratakis 2017-05-29 13:17:45 EDT
*** Bug 1319963 has been marked as a duplicate of this bug. ***
Comment 21 Carl George 2017-05-29 14:46:22 EDT
Thank you Charalampos.
Comment 22 Fedora Update System 2017-05-30 18:05:57 EDT
python34-3.4.5-4.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-1e0d75a462
Comment 23 Fedora Update System 2017-06-02 15:18:43 EDT
python34-3.4.5-4.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.