Bug 1599422

Summary: Time to have virtualenv run by Python 3?
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: python-virtualenvAssignee: Miro Hrončok <mhroncok>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: cstratak, fschwarz, jeremy, j, mcyprian, mhayden, mrunge, ncoghlan, ngompa13, pviktori, python-maint, python-sig, smilner
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-virtualenv-16.0.0-5.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-27 13:45:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Miro Hrončok 2018-07-09 18:16:16 UTC
Currently, /usr/bin/virtualenv is shipped from the python2-virtualenv package and is being run by /usr/bin/python2.

This is in fact very much OK with the current guidelines:

https://fedoraproject.org/wiki/Packaging:Python#Executables_in_.2Fusr.2Fbin

> If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.

> If executables are to be shipped for both python 2 and python 3 ... The unversioned executable must be the python2 version.


However this is also problematic:

If we want to ship the "virtualenv" command in Python Classroom Lab, we need to ship Python 2. We don't want tot ship Python 2 in environment that's focusing on teaching, as it teaches wrong habits.

If we however only ship python3-virtualenv, users won't find the virtualenv command where they expect it. Yes, they can use the venv module from the standard library but I'd still like the virtualenv command to be available.


The entire "provide different functionality" here is that by default, virtualenv creates virtual environments with Python that virtualenv is being executed by:

  -p PYTHON_EXE, --python=PYTHON_EXE
                        The Python interpreter to use, e.g.,
                        --python=python3.5 will use the python3.5 interpreter
                        to create the new environment.  The default is the
                        interpreter that virtualenv was installed with
                        (/usr/bin/python2)

vs.

  -p PYTHON_EXE, --python=PYTHON_EXE
                        The Python interpreter to use, e.g.,
                        --python=python3.5 will use the python3.5 interpreter
                        to create the new environment.  The default is the
                        interpreter that virtualenv was installed with
                        (/usr/bin/python3)


I think it would be nice to have this changed in a way that /usr/bin/virtualenv is run by python3.


There are several options I see here:

1) break the rules (or ask FPC for an exception) and make virtualev be python3 based, while preserving the versioned executables for backwards compatibility

2) say the behavior is the same, ignore the default -p value, ship only one executable and call it a day (meh, I don't like this very much)

3) patch virtualenv so that the default is not determined by what python runs this but rather some creepy logic that takes in the executable name and defaults to python2 if it is "virtualenv" (I don't like diverging from upstream very much*), bah "the behavior is the same" now, virtualenv can switch to py3, versioned executables are shipped for backwards compatibility only

4) make /usr/bin/virtualenv a shell script that calls virtualenv-2 if available, virtualenv-3 if not, ship it from both subpackages (add explicit conflicts for smaller versions of the other subpackage)

* an upstream change that respects some /etc/ config file first and only fallbacks to the current behavior would be possible, yet maybe deemed too complicated by upstream


The question is, whether user expectations are that "virtualenv foo" will crate a python2 virtualenv. I've always only used virtualenv with explicit --python switch, so I have no idea. If this expectation is not strong, I'd go with 1) (asking FPC). If the expectation about this behavior is strong, I'd go with 3) or 4), whatever is less fragile.

Comment 1 Neal Gompa 2018-07-10 11:44:51 UTC
I thought we had switched the guidelines a while ago to have unversioned binaries be Python 3. I know we've talked about it for years now... Did it really not change yet?

Comment 2 Miro Hrončok 2018-07-10 11:56:12 UTC
Unfortunately not. One of the (valid) arguments is that since python command is python2, unversioned commands should also be python2. With pip, I kinda get it, but there are other use cases, where I don't like it that much.

Comment 3 Petr Viktorin (pviktori) 2018-07-10 13:51:05 UTC
What about not shipping virtualenv at all, and telling people to use `python3 -m venv`?

Comment 4 Miro Hrončok 2018-07-10 14:44:05 UTC
We can decide not to ship virtualenv in the Python Classroom Lab. However where and how do we tell people to use venv?

Comment 5 Steve Milner 2018-07-10 14:55:08 UTC
(In reply to Petr Viktorin from comment #3)
> What about not shipping virtualenv at all, and telling people to use
> `python3 -m venv`?

FWIW that command doesn't seem to work across different Fedora spins. Here's an example on Silverblue:

$ python3 -m venv testing
Error: Command '['/tmp/testing/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
$ python3 --version
Python 3.6.5

(In reply to Miro Hrončok from comment #4)
> We can decide not to ship virtualenv in the Python Classroom Lab. However
> where and how do we tell people to use venv?

To be fair I'm not sure if python3 -m venv meets all the requirements for virtualenv users or not. *IF* it does *AND* the community would like the move to using venv instead of virtualenv there are a few things that could be done:

- pythonX-virtualenv becomes a script which warns about the command being removed in favor of python3 -m venv. In the meantime it forwards to python3 -m venv $ARGS.
- python3 package is upadted with a virtualenv script which forwards to python3 -m venv $ARGS. python3 is updated to provide what the virtualenv package(s) provided.

Comment 6 Miro Hrončok 2018-07-10 15:00:02 UTC
$ python3 -m venv testing
Error: Command '['/tmp/testing/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
$ python3 --version
Python 3.6.5

This should not happen if shipped form Fedora RPMs. I would consider it a bug.

Comment 7 Neal Gompa 2018-07-10 15:02:49 UTC
(In reply to Miro Hrončok from comment #2)
> Unfortunately not. One of the (valid) arguments is that since python command
> is python2, unversioned commands should also be python2. With pip, I kinda
> get it, but there are other use cases, where I don't like it that much.

No one else is doing it this way. They're either fully versioned with alternatives managing the unversioned name (openSUSE, Debian), or they're only shipping the Python 3 version as the unversioned binary (OpenMandriva, Mageia, Fedora[ish]).

Comment 8 Miro Hrončok 2018-07-10 15:03:14 UTC
(In reply to Steve Milner from comment #5)
> To be fair I'm not sure if python3 -m venv meets all the requirements for
> virtualenv users or not. 

IMHO it does not. virtualenv works for older Pythons as well. It can run on new Python versions and let you decide what python version (or rather executable) you want to use inside the venv.

-m venv only works with newer versions and will always create a venv for the Python version (executable) that runs it.

Also, AFAIK virtualenv is used internally by other tools such as tox.

Comment 9 Petr Viktorin (pviktori) 2018-07-10 15:04:37 UTC
(In reply to Miro Hrončok from comment #4)
> We can decide not to ship virtualenv in the Python Classroom Lab. However
> where and how do we tell people to use venv?

It already is on the Fedora Developer Portal.
I guess there are no release notes specifically for Classroom, right?

(In reply to Steve Milner from comment #5)
> (In reply to Petr Viktorin from comment #3)
> > What about not shipping virtualenv at all, and telling people to use
> > `python3 -m venv`?
> 
> FWIW that command doesn't seem to work across different Fedora spins. Here's
> an example on Silverblue:
> 
> $ python3 -m venv testing
> Error: Command '['/tmp/testing/bin/python3', '-Im', 'ensurepip',
> '--upgrade', '--default-pip']' returned non-zero exit status 1.
> $ python3 --version
> Python 3.6.5

That's quite bad.
What's Silverblue and where do I file a bug for it?


> (In reply to Miro Hrončok from comment #4)
> > We can decide not to ship virtualenv in the Python Classroom Lab. However
> > where and how do we tell people to use venv?
> 
> To be fair I'm not sure if python3 -m venv meets all the requirements for
> virtualenv users or not. *IF* it does *AND* the community would like the
> move to using venv instead of virtualenv there are a few things that could
> be done:
> 
> - pythonX-virtualenv becomes a script which warns about the command being
> removed in favor of python3 -m venv. In the meantime it forwards to python3
> -m venv $ARGS.
> - python3 package is upadted with a virtualenv script which forwards to
> python3 -m venv $ARGS. python3 is updated to provide what the virtualenv
> package(s) provided.

When functionality moves from 3rd party packages to the standard library, it's usually cleaned up and simplified a bit. The virtualenv -> venv transition is no exception.
However, it does meet the vast majority of use cases.
The main thing that's different is the Python API.

Comment 10 Steve Milner 2018-07-10 17:56:08 UTC
(In reply to Petr Viktorin from comment #9) 
> That's quite bad.
> What's Silverblue and where do I file a bug for it?

"Silverblue is the new face of Fedora Atomic Workstation from Project Atomic. With good support for container-focused workflows, this variant of Fedora Workstation targets developer communities. If you want to emphasize that it is part of the Fedora project, calling it Fedora Silverblue is fine, too."

Site: https://teamsilverblue.org/
Issues: https://pagure.io/teamsilverblue/issues

Comment 11 Petr Viktorin (pviktori) 2018-07-11 08:26:37 UTC
Silverblue failure reported at https://pagure.io/teamsilverblue/issue/21 . It's an unrelated bug; please discuss it there.

Comment 12 Miro Hrončok 2018-08-10 14:54:35 UTC
OK. No more discussions. Should we go with option 2 then?

Comment 13 Jan Kurik 2018-08-14 09:54:55 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

Comment 14 Steve Milner 2018-08-15 20:48:36 UTC
> 4) make /usr/bin/virtualenv a shell script that calls virtualenv-2 if available, virtualenv-3 if not, ship it from both subpackages (add explicit conflicts for smaller versions of the other subpackage)

Why not invert this? A shell script that calls virtualenv-3 if available. If it isn't, it falls back to the older virtualenv-2? I'd also be cool with using alternatives if people feel strongly about that.

Comment 15 Miro Hrončok 2018-08-16 10:07:22 UTC
I f we prefer virtualenv-3 we might as well go with option 2 and save ourselves some work and diverging upstream.

Comment 16 Steve Milner 2018-08-20 19:46:31 UTC
Then let's run with option 2. I'm a little worried about pushing people towards python 2 by default.

Comment 17 Miro Hrončok 2018-08-23 17:51:12 UTC
I can do this, but I'd rather get https://src.fedoraproject.org/rpms/python-virtualenv/pull-request/4 merged first so I don't create conflicts.