Bug 2165702

Summary: python3-virtualenv needs to add RPM Requires for python3.11-wheel-wheel and be tested with Python 3.11 OR it needs to clearly communicate it doesn't work
Product: Red Hat Enterprise Linux 8 Reporter: Miro Hrončok <mhroncok>
Component: python36-3.6-moduleAssignee: Python Maintainers <python-maint>
Status: CLOSED ERRATA QA Contact: Lukáš Zachar <lzachar>
Severity: unspecified Docs Contact: Lenka Špačková <lkuprova>
Priority: unspecified    
Version: CentOS StreamCC: bstinson, jwboyer, maxwell, pviktori, python-maint, torsava
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-virtualenv-15.1.0-22.module+el8.8.0+18131+2eb9cbd8 Doc Type: Known Issue
Doc Text:
.Creating virtual Python 3.11 environments fails when using the `virtualenv` utility The `virtualenv` utility in RHEL 8, provided by the `python3-virtualenv` package, is not compatible with Python 3.11. An attempt to create a virtual environment by using `virtualenv` will fail with the following error message: ---- $ virtualenv -p python3.11 venv3.11 Running virtualenv with interpreter /usr/bin/python3.11 ERROR: Virtual environments created by virtualenv < 20 are not compatible with Python 3.11. ERROR: Use `python3.11 -m venv` instead. ---- To create Python 3.11 virtual environments, use the `python3.11 -m venv` command instead, which uses the `venv` module from the standard library.
 Story Points: ---
Clone Of:
: 2168662 (view as bug list) Environment:
Last Closed: 2023-05-16 08:43:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2165872    
Bug Blocks: 2137139, 2168662    

Description Miro Hrončok 2023-01-30 19:37:42 UTC 
python3-virtualenv from c8(s)-stream-3.6 has:

  # RPM installed wheels
  Requires:       python3-pip-wheel
  Requires:       python3-setuptools-wheel
  Requires:       python3-wheel-wheel
  Requires:       (python2-wheel-wheel if python2)
  Requires:       (python38-wheel-wheel if python38)
  Requires:       (python39-wheel-wheel if python39)

With the addition of Python 3.11, it needs to gain:

  Requires:       (python3.11-wheel-wheel if python3.11)

I also assume virtualenv 15.1.0 might have some trouble creating virtual environments for Python 3.11, so this needs to be tested (and possibly patched).


c8(s)-stream-2.7 appears to also have python2-virtualenv -- same changes need to be done there :(


Other streams don't seem to have virtualenv, according to https://git.centos.org/rpms/python-virtualenv/branches
Comment 2 Miro Hrončok 2023-02-01 14:26:51 UTC 
$ rpm -q python3-virtualenv
python3-virtualenv-15.1.0-21.module_el8.5.0+2569+5c5719bc.noarch

$ rpm -qa | grep python3.11
python3.11-pip-wheel-22.3.1-2.el8.noarch
python3.11-libs-3.11.1-4.el8.x86_64
python3.11-wheel-wheel-0.38.4-3.el8.noarch
python3.11-setuptools-wheel-65.5.1-2.el8.noarch
python3.11-3.11.1-4.el8.x86_64
python3.11-devel-3.11.1-4.el8.x86_64


$ virtualenv -p python3.11 venv3.11
Running virtualenv with interpreter /usr/bin/python3.11
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.sysconfig
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  import distutils.sysconfig
Using base prefix '/usr'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 2376, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 705, in main
    create_environment(home_dir,
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 933, in create_environment
    py_executable = os.path.abspath(install_python(
                                    ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 1158, in install_python
    fix_lib64(lib_dir, symlink)
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 1578, in fix_lib64
    assert os.path.basename(lib_dir) == 'python%s' % sys.version[:3], (
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Unexpected python lib dir: '/builddir/venv3.11/lib/python3.11'


OK, this indeed does not work at all.
Comment 3 Miro Hrončok 2023-02-01 14:33:01 UTC 
https://github.com/pypa/virtualenv/pull/1385 adds support for double-digit Python versions.
Comment 5 Miro Hrončok 2023-02-01 16:15:21 UTC 
I've tried backporting the following three commits:

 - https://github.com/pypa/virtualenv/pull/1064
 - https://github.com/pypa/virtualenv/pull/1385
 - https://github.com/pypa/virtualenv/pull/2109

It makes this old virtualenv create Python 3.10 environments \o/ but not 3.11 :(

I've checked and the last virtualenv upstream version < 20 behaves the same, so it is unlikely there will be anything else to backport.


The current error is:

$ virtualenv -p python3.11 venv3.11 --no-download --verbose 
Running virtualenv with interpreter /usr/bin/python3.11
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.sysconfig
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  import distutils.sysconfig
Using base prefix '/usr'
Creating /builddir/venv3.11/lib/python3.11
Symlinking Python bootstrap modules
  Symlinking /builddir/venv3.11/lib/python3.11/config-3.11-x86_64-linux-gnu
  Symlinking /builddir/venv3.11/lib/python3.11/lib-dynload
  Cannot import bootstrap module: os
  Ignoring built-in bootstrap module: posix
  Cannot import bootstrap module: posixpath
  Cannot import bootstrap module: nt
  Cannot import bootstrap module: ntpath
  Cannot import bootstrap module: genericpath
  Symlinking /builddir/venv3.11/lib64/python3.11/fnmatch.py
  Symlinking /builddir/venv3.11/lib64/python3.11/locale.py
  Symlinking /builddir/venv3.11/lib64/python3.11/encodings
  Cannot import bootstrap module: codecs
  Cannot import bootstrap module: stat
  Cannot import bootstrap module: UserDict
  File /builddir/venv3.11/lib64/python3.11/lib-dynload/readline.cpython-311-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: copy_reg
  Symlinking /builddir/venv3.11/lib64/python3.11/types.py
  Symlinking /builddir/venv3.11/lib64/python3.11/re
  Cannot import bootstrap module: sre
  Symlinking /builddir/venv3.11/lib64/python3.11/sre_parse.py
  Symlinking /builddir/venv3.11/lib64/python3.11/sre_constants.py
  Symlinking /builddir/venv3.11/lib64/python3.11/sre_compile.py
  File /builddir/venv3.11/lib64/python3.11/lib-dynload/zlib.cpython-311-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: _abcoll
  Symlinking /builddir/venv3.11/lib64/python3.11/warnings.py
  Symlinking /builddir/venv3.11/lib64/python3.11/linecache.py
  Cannot import bootstrap module: abc
  Cannot import bootstrap module: io
  Symlinking /builddir/venv3.11/lib64/python3.11/_weakrefset.py
  Symlinking /builddir/venv3.11/lib64/python3.11/copyreg.py
  Symlinking /builddir/venv3.11/lib64/python3.11/tempfile.py
  Symlinking /builddir/venv3.11/lib64/python3.11/random.py
  Symlinking /builddir/venv3.11/lib64/python3.11/__future__.py
  Symlinking /builddir/venv3.11/lib64/python3.11/collections
  Symlinking /builddir/venv3.11/lib64/python3.11/keyword.py
  Symlinking /builddir/venv3.11/lib64/python3.11/tarfile.py
  Symlinking /builddir/venv3.11/lib64/python3.11/shutil.py
  Symlinking /builddir/venv3.11/lib64/python3.11/struct.py
  Symlinking /builddir/venv3.11/lib64/python3.11/copy.py
  Symlinking /builddir/venv3.11/lib64/python3.11/tokenize.py
  Symlinking /builddir/venv3.11/lib64/python3.11/token.py
  Symlinking /builddir/venv3.11/lib64/python3.11/functools.py
  Symlinking /builddir/venv3.11/lib64/python3.11/heapq.py
  Symlinking /builddir/venv3.11/lib64/python3.11/bisect.py
  Symlinking /builddir/venv3.11/lib64/python3.11/weakref.py
  Symlinking /builddir/venv3.11/lib64/python3.11/reprlib.py
  Symlinking /builddir/venv3.11/lib64/python3.11/base64.py
  Cannot import bootstrap module: _dummy_thread
  Symlinking /builddir/venv3.11/lib64/python3.11/hashlib.py
  Symlinking /builddir/venv3.11/lib64/python3.11/hmac.py
  Symlinking /builddir/venv3.11/lib64/python3.11/imp.py
  Symlinking /builddir/venv3.11/lib64/python3.11/importlib
  Symlinking /builddir/venv3.11/lib64/python3.11/rlcompleter.py
  Symlinking /builddir/venv3.11/lib64/python3.11/operator.py
  Cannot import bootstrap module: _collections_abc
  Cannot import bootstrap module: _bootlocale
  Symlinking /builddir/venv3.11/lib64/python3.11/enum.py
Creating /builddir/venv3.11/lib/python3.11/site-packages
Writing /builddir/venv3.11/lib64/python3.11/site.py
Writing /builddir/venv3.11/lib64/python3.11/orig-prefix.txt
Writing /builddir/venv3.11/lib64/python3.11/no-global-site-packages.txt
Creating parent directories for /builddir/venv3.11/include
Symlinking /builddir/venv3.11/include/python3.11
Creating /builddir/venv3.11/bin
New python executable in /builddir/venv3.11/bin/python3.11
Changed mode of /builddir/venv3.11/bin/python3.11 to 0o755
Also creating executable in /builddir/venv3.11/bin/python
Changed mode of /builddir/venv3.11/bin/python to 0o755
Testing executable with /builddir/venv3.11/bin/python3.11 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
ERROR: The executable /builddir/venv3.11/bin/python3.11 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/builddir/venv3.11')
ERROR: virtualenv is not compatible with this system or executable



My best guess is that this code:

        filename = find_module_filename(modname)
        if filename is None:
            logger.info("Cannot import bootstrap module: %s" % modname)

in copy_required_modules() is chocking on https://docs.python.org/3.11/whatsnew/3.11.html#frozen-imports-static-code-objects


$ python3.6
...
>>> from virtualenv import find_module_filename
>>> find_module_filename('os')
'/usr/lib64/python3.6/os.py'

$ python3.10
...

>>> from virtualenv import find_module_filename
>>> find_module_filename('os')
'/usr/lib64/python3.10/os.py'

$ python3.11
...
>>> from virtualenv import find_module_filename
>>> find_module_filename('os')
(None)


I will stop for now and get back to this problem later.
Comment 6 Miro Hrončok 2023-02-01 18:12:06 UTC 
I've managed to solve that problem with:

        if not spec.has_location and spec.origin == "frozen":
            # https://docs.python.org/3.11/whatsnew/3.11.html#frozen-imports-static-code-objects
            # For frozen modules, we assume they are from /usr/lib64/python3.X/<name>.py
            import sysconfig  # always available on Python 3.11+ which has frozen modules
            return os.path.join(sysconfig.get_path("stdlib"), modname + ".py")


But now apparently venv3.11/bin/python isn't loading venv3.11/lib/python3.11/site.py when being executed and hence sys.path is missing the actual standard library.


$ venv3.6/bin/python -c 'import sys; print(sys.path)'
['', '/builddir/venv3.6/lib64/python36.zip', '/builddir/venv3.6/lib64/python3.6', '/builddir/venv3.6/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6', '/usr/lib/python3.6', '/builddir/venv3.6/lib/python3.6/site-packages']

$ venv3.11/bin/python -c 'import sys; print(sys.path)'
['', '/builddir/venv3.11/lib64/python311.zip', '/builddir/venv3.11/lib64/python3.11', '/builddir/venv3.11/lib64/python3.11/lib-dynload', '/builddir/venv3.11/lib64/python3.11/site-packages', '/builddir/venv3.11/lib/python3.11/site-packages']




Adding raise RuntimeError("bah") to venv3.X/lib/python3.X/site.py:

$ venv3.6/bin/python -c pass
Failed to import the site module
Traceback (most recent call last):
  File "/builddir/venv3.6/lib64/python3.6/site.py", line 66, in <module>
    raise RuntimeError("bah")
RuntimeError: bah

$ venv3.11/bin/python -c pass
(nothing)


That's... unfortunate.
Comment 7 Miro Hrončok 2023-02-02 11:10:38 UTC 
I have a working patch that does:

  $ virtualenv -p python3.11 venv3.11 
  Running virtualenv with interpreter /usr/bin/python3.11
  ERROR: Virtual environments created by virtualenv < 20 are not compatible with Python 3.11.
  ERROR: Use `python3.11 -m venv` instead.
  (exit code 1)

I believe this might be the most sensible approach this late in the RHEL 8 release cycle. Note that 3.12+ would need to be ported away from distutils anyway.
Comment 23 Petr Viktorin (pviktori) 2023-02-27 08:30:56 UTC 
> Please review the proposed note.

I'd prefer not calling virtualenv “legacy”. It's only legacy in RHEL -- it has a few more features, but not enough to justify Red Hat supporting it in the future.
Upstream, it's actively maintained.
I suggest dropping the word:

> .Creating virtual Python 3.11 environments fails when using the `virtualenv` utility

or maybe more positively:

> .Python 3.11 virtual environments should be created using `venv`, rather than the `virtualenv` utility

And the next sentence doesn't look correct. Virtual environments are specific to a version of Python. Pre-existing ones are not compatible with 3.11, but also with any version other that the one they were made for.
I suggest:

> The `virtualenv` utility in RHEL 8, provided by the `python3-virtualenv` package, is not compatible with Python 3.11. An attempt to use it to create a virtual environment will fail with the following error message:
Comment 24 Miro Hrončok 2023-02-27 08:49:32 UTC 
My understanding was that the "legacy" relates to the old version we ship, not to the tool itself. But it could be understood both ways and the adjustments by Petr make sense.
Comment 28 errata-xmlrpc 2023-05-16 08:43:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (python36:3.6 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2916