Bug 2011455

Summary:	Recent change in Python sysconfig install scheme broke virtualenv
Product:	[Fedora] Fedora	Reporter:	Miro Hrončok <mhroncok>
Component:	python-virtualenv	Assignee:	Miro Hrončok <mhroncok>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	cstratak, lbalhar, m.cyprian, mhroncok, mrunge, python-sig, smilner
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-virtualenv-20.8.1-1.fc36	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-07 13:18:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Miro Hrončok 2021-10-06 15:48:12 UTC

See https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/

This change apparently broke virtualenv:

Before:

$ virtualenv --python=python3.10 venv
created virtual environment CPython3.10.0.final.0-64 in 158ms
  creator CPython3Posix(dest=/builddir/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(extra_search_dir=/usr/share/python-wheels,download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/builddir/.local/share/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
$ ls venv/bin/
activate  activate.csh  activate.fish  activate.ps1  activate_this.py  pip  pip-3.10  pip3  pip3.10  python  python3  python3.10  wheel  wheel-3.10  wheel3  wheel3.10


After:

$ virtualenv --python=python3.10 venv
created virtual environment CPython3.10.0.final.0-64 in 200ms
  creator CPython3Posix(dest=/builddir/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(extra_search_dir=/usr/share/python-wheels,download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/builddir/.local/share/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator

$ ls venv/
lib64  local  pyvenv.cfg


There is no error message, no bad exit code, nothing.


python-setuptools-wheel-57.4.0-1.fc35.noarch
python-wheel-wheel-0.37.0-1.fc36.noarch
python3-distlib-0.3.2-1.fc35.noarch
python3-filelock-3.0.12-12.fc35.noarch
python3-platformdirs-2.3.0-1.fc36.noarch
python3-six-1.16.0-4.fc35.noarch
python3-virtualenv-20.7.2-1.fc36.noarch
python-pip-wheel-21.2.3-3.fc36.noarch
python3-3.10.0-2.fc36.x86_64
python3-libs-3.10.0-2.fc36.x86_64

Comment 1 Miro Hrončok 2021-10-06 17:15:51 UTC

$ virtualenv --python=python3.10 venv2 -vvvv
55 setup logging to NOTSET [DEBUG report:43]
58 find interpreter for spec PythonSpec(implementation=CPython, major=3, minor=10) [INFO builtin:62]
58 proposed PythonInfo(spec=CPython3.10.0.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.0 (default, Oct  5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:69]
58 accepted PythonInfo(spec=CPython3.10.0.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.0 (default, Oct  5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:71]
59 filesystem is case-sensitive [DEBUG info:28]
92 create virtual environment via CPython3Posix(dest=/builddir/venv2, clear=False, no_vcs_ignore=False, global=False) [INFO session:52]
92 create folder /builddir/venv2/lib64/python3.10 [DEBUG _sync:25]
93 create folder /builddir/venv2/local/bin [DEBUG _sync:25]
93 create folder /builddir/venv2/local/lib/python3.10/site-packages [DEBUG _sync:25]
93 create folder /builddir/venv2/local/lib64/python3.10/site-packages [DEBUG _sync:25]
93 write /builddir/venv2/pyvenv.cfg [DEBUG pyenv_cfg:34]
93 	home = /usr [DEBUG pyenv_cfg:38]
93 	implementation = CPython [DEBUG pyenv_cfg:38]
93 	version_info = 3.10.0.final.0 [DEBUG pyenv_cfg:38]
93 	virtualenv = 20.7.2 [DEBUG pyenv_cfg:38]
93 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
93 	base-prefix = /usr [DEBUG pyenv_cfg:38]
93 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
93 	base-executable = /usr/bin/python3 [DEBUG pyenv_cfg:38]
93 symlink /usr/bin/python3 to /builddir/venv2/local/bin/python [DEBUG _sync:44]
94 create virtualenv import hook file /builddir/venv2/local/lib/python3.10/site-packages/_virtualenv.pth [DEBUG api:95]
94 create /builddir/venv2/local/lib/python3.10/site-packages/_virtualenv.py [DEBUG api:98]
94 ============================== target debug ============================== [DEBUG session:54]
94 debug via /builddir/venv2/local/bin/python /usr/lib/python3.10/site-packages/virtualenv/create/debug.py [DEBUG creator:224]
94 {
  "sys": {
    "executable": "/builddir/venv2/local/bin/python",
    "_base_executable": "/builddir/venv2/local/bin/python",
    "prefix": "/usr",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/usr",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib64/python310.zip",
      "/usr/lib64/python3.10",
      "/usr/lib64/python3.10/lib-dynload",
      "/usr/lib64/python3.10/site-packages",
      "/usr/lib/python3.10/site-packages"
    ],
    "meta_path": [
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.0 (default, Oct  5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]",
  "makefile_filename": "/usr/lib64/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib64/python3.10/os.py'>",
  "site": "<module 'site' from '/usr/lib64/python3.10/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib64/python3.10/datetime.py'>",
  "math": "<module 'math' from '/usr/lib64/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/usr/lib64/python3.10/json/__init__.py'>"
} [DEBUG session:55]
145 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/builddir/.local/share/virtualenv) [INFO session:59]
199 install setuptools from wheel /usr/share/python-wheels/setuptools-57.4.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:50]
199 install wheel from wheel /usr/share/python-wheels/wheel-0.37.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:50]
200 Attempting to acquire lock 140433895328976 on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [DEBUG filelock:270]
200 Lock 140433895328976 acquired on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [INFO filelock:274]
200 Attempting to acquire lock 140433895326144 on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [DEBUG filelock:270]
200 install pip from wheel /usr/share/python-wheels/pip-21.2.3-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:50]
200 Lock 140433895326144 acquired on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [INFO filelock:274]
200 Attempting to release lock 140433895328976 on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [DEBUG filelock:315]
201 Lock 140433895328976 released on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any.lock [INFO filelock:318]
201 Attempting to release lock 140433895326144 on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [DEBUG filelock:315]
201 Lock 140433895326144 released on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any.lock [INFO filelock:318]
201 copy /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/distutils-precedence.pth to /builddir/venv2/local/lib/python3.10/site-packages/distutils-precedence.pth [DEBUG _sync:52]
201 copy /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel-0.37.0.virtualenv to /builddir/venv2/local/lib/python3.10/site-packages/wheel-0.37.0.virtualenv [DEBUG _sync:52]
201 Attempting to acquire lock 140433895326144 on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [DEBUG filelock:270]
201 Lock 140433895326144 acquired on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [INFO filelock:274]
202 Attempting to release lock 140433895326144 on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [DEBUG filelock:315]
202 Lock 140433895326144 released on /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any.lock [INFO filelock:318]
202 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools to /builddir/venv2/local/lib/python3.10/site-packages/setuptools [DEBUG _sync:52]
202 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel-0.37.0.dist-info to /builddir/venv2/local/lib/python3.10/site-packages/wheel-0.37.0.dist-info [DEBUG _sync:52]
202 copy /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip-21.2.3.virtualenv to /builddir/venv2/local/lib/python3.10/site-packages/pip-21.2.3.virtualenv [DEBUG _sync:52]
203 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip to /builddir/venv2/local/lib/python3.10/site-packages/pip [DEBUG _sync:52]
206 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.0-py2.py3-none-any/wheel to /builddir/venv2/local/lib/python3.10/site-packages/wheel [DEBUG _sync:52]
215 generated console scripts wheel3 wheel3.10 wheel-3.10 wheel [DEBUG base:48]
237 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/_distutils_hack to /builddir/venv2/local/lib/python3.10/site-packages/_distutils_hack [DEBUG _sync:52]
237 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/pkg_resources to /builddir/venv2/local/lib/python3.10/site-packages/pkg_resources [DEBUG _sync:52]
242 copy /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools-57.4.0.virtualenv to /builddir/venv2/local/lib/python3.10/site-packages/setuptools-57.4.0.virtualenv [DEBUG _sync:52]
242 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-57.4.0-py3-none-any/setuptools-57.4.0.dist-info to /builddir/venv2/local/lib/python3.10/site-packages/setuptools-57.4.0.dist-info [DEBUG _sync:52]
244 generated console scripts  [DEBUG base:48]
267 copy directory /builddir/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-21.2.3-py3-none-any/pip-21.2.3.dist-info to /builddir/venv2/local/lib/python3.10/site-packages/pip-21.2.3.dist-info [DEBUG _sync:52]
269 generated console scripts pip3 pip pip3.10 pip-3.10 [DEBUG base:48]
269 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
270 write /builddir/venv2/pyvenv.cfg [DEBUG pyenv_cfg:34]
271 	home = /usr [DEBUG pyenv_cfg:38]
271 	implementation = CPython [DEBUG pyenv_cfg:38]
271 	version_info = 3.10.0.final.0 [DEBUG pyenv_cfg:38]
271 	virtualenv = 20.7.2 [DEBUG pyenv_cfg:38]
271 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
271 	base-prefix = /usr [DEBUG pyenv_cfg:38]
271 	base-exec-prefix = /usr [DEBUG pyenv_cfg:38]
271 	base-executable = /usr/bin/python3 [DEBUG pyenv_cfg:38]
271 created virtual environment CPython3.10.0.final.0-64 in 217ms
  creator CPython3Posix(dest=/builddir/venv2, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(extra_search_dir=/usr/share/python-wheels,download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/builddir/.local/share/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]

Comment 2 Miro Hrončok 2021-10-06 17:49:53 UTC

I've opened an upstream virtualenv issue https://github.com/pypa/virtualenv/issues/2208

However, I think this is on us :(

A hacky workaround is to hotpatch virtualenv/discovery/py_info.py PythonInfo.install_path():

        # A hack for https://github.com/pypa/virtualenv/issues/2208
        if result.startswith(u"local/"):
            return result[6:]
        return result

Comment 3 Miro Hrončok 2021-10-06 18:32:28 UTC

*** Bug 2011475 has been marked as a duplicate of this bug. ***

Comment 4 Miro Hrončok 2021-10-06 18:43:37 UTC

https://src.fedoraproject.org/rpms/python-virtualenv/pull-request/48

Comment 5 Fedora Update System 2021-10-07 13:16:25 UTC

FEDORA-2021-642854e1cc has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2021-642854e1cc

Comment 6 Fedora Update System 2021-10-07 13:18:57 UTC

FEDORA-2021-642854e1cc has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.