RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2165702 - python3-virtualenv needs to add RPM Requires for python3.11-wheel-wheel and be tested with Python 3.11 OR it needs to clearly communicate it doesn't work
Summary: python3-virtualenv needs to add RPM Requires for python3.11-wheel-wheel and b...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: python36-3.6-module
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Python Maintainers
QA Contact: Lukáš Zachar
Lenka Špačková
URL:
Whiteboard:
Depends On: 2165872
Blocks: 2137139 2168662
TreeView+ depends on / blocked
 
Reported: 2023-01-30 19:37 UTC by Miro Hrončok
Modified: 2023-05-16 10:12 UTC (History)
6 users (show)

Fixed In Version: python-virtualenv-15.1.0-22.module+el8.8.0+18131+2eb9cbd8
Doc Type: Known Issue
Doc Text:
.Creating virtual Python 3.11 environments fails when using the `virtualenv` utility The `virtualenv` utility in RHEL 8, provided by the `python3-virtualenv` package, is not compatible with Python 3.11. An attempt to create a virtual environment by using `virtualenv` will fail with the following error message: ---- $ virtualenv -p python3.11 venv3.11 Running virtualenv with interpreter /usr/bin/python3.11 ERROR: Virtual environments created by virtualenv < 20 are not compatible with Python 3.11. ERROR: Use `python3.11 -m venv` instead. ---- To create Python 3.11 virtual environments, use the `python3.11 -m venv` command instead, which uses the `venv` module from the standard library.
Clone Of:
: 2168662 (view as bug list)
Environment:
Last Closed: 2023-05-16 08:43:05 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github pypa virtualenv pull 1064 0 None Merged Remove universal newline flag 2023-02-01 16:15:21 UTC
Github pypa virtualenv pull 1385 0 None Merged Fixes for python 3.10 2023-02-01 14:33:01 UTC
Github pypa virtualenv pull 2109 0 None Merged Legacy: Use sysconfig.get_default_scheme() where available 2023-02-01 16:15:21 UTC
Red Hat Issue Tracker RHELPLAN-146937 0 None None None 2023-01-30 19:38:09 UTC
Red Hat Product Errata RHBA-2023:2916 0 None None None 2023-05-16 08:43:12 UTC

Description Miro Hrončok 2023-01-30 19:37:42 UTC 
python3-virtualenv from c8(s)-stream-3.6 has:

  # RPM installed wheels
  Requires:       python3-pip-wheel
  Requires:       python3-setuptools-wheel
  Requires:       python3-wheel-wheel
  Requires:       (python2-wheel-wheel if python2)
  Requires:       (python38-wheel-wheel if python38)
  Requires:       (python39-wheel-wheel if python39)

With the addition of Python 3.11, it needs to gain:

  Requires:       (python3.11-wheel-wheel if python3.11)

I also assume virtualenv 15.1.0 might have some trouble creating virtual environments for Python 3.11, so this needs to be tested (and possibly patched).


c8(s)-stream-2.7 appears to also have python2-virtualenv -- same changes need to be done there :(


Other streams don't seem to have virtualenv, according to https://git.centos.org/rpms/python-virtualenv/branches
Comment 2 Miro Hrončok 2023-02-01 14:26:51 UTC 
$ rpm -q python3-virtualenv
python3-virtualenv-15.1.0-21.module_el8.5.0+2569+5c5719bc.noarch

$ rpm -qa | grep python3.11
python3.11-pip-wheel-22.3.1-2.el8.noarch
python3.11-libs-3.11.1-4.el8.x86_64
python3.11-wheel-wheel-0.38.4-3.el8.noarch
python3.11-setuptools-wheel-65.5.1-2.el8.noarch
python3.11-3.11.1-4.el8.x86_64
python3.11-devel-3.11.1-4.el8.x86_64


$ virtualenv -p python3.11 venv3.11
Running virtualenv with interpreter /usr/bin/python3.11
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.sysconfig
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  import distutils.sysconfig
Using base prefix '/usr'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 2376, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 705, in main
    create_environment(home_dir,
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 933, in create_environment
    py_executable = os.path.abspath(install_python(
                                    ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 1158, in install_python
    fix_lib64(lib_dir, symlink)
  File "/usr/lib/python3.6/site-packages/virtualenv.py", line 1578, in fix_lib64
    assert os.path.basename(lib_dir) == 'python%s' % sys.version[:3], (
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Unexpected python lib dir: '/builddir/venv3.11/lib/python3.11'


OK, this indeed does not work at all.
Comment 3 Miro Hrončok 2023-02-01 14:33:01 UTC 
https://github.com/pypa/virtualenv/pull/1385 adds support for double-digit Python versions.
Comment 5 Miro Hrončok 2023-02-01 16:15:21 UTC 
I've tried backporting the following three commits:

 - https://github.com/pypa/virtualenv/pull/1064
 - https://github.com/pypa/virtualenv/pull/1385
 - https://github.com/pypa/virtualenv/pull/2109

It makes this old virtualenv create Python 3.10 environments \o/ but not 3.11 :(

I've checked and the last virtualenv upstream version < 20 behaves the same, so it is unlikely there will be anything else to backport.


The current error is:

$ virtualenv -p python3.11 venv3.11 --no-download --verbose 
Running virtualenv with interpreter /usr/bin/python3.11
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.sysconfig
/usr/lib/python3.6/site-packages/virtualenv.py:25: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  import distutils.sysconfig
Using base prefix '/usr'
Creating /builddir/venv3.11/lib/python3.11
Symlinking Python bootstrap modules
  Symlinking /builddir/venv3.11/lib/python3.11/config-3.11-x86_64-linux-gnu
  Symlinking /builddir/venv3.11/lib/python3.11/lib-dynload
  Cannot import bootstrap module: os
  Ignoring built-in bootstrap module: posix
  Cannot import bootstrap module: posixpath
  Cannot import bootstrap module: nt
  Cannot import bootstrap module: ntpath
  Cannot import bootstrap module: genericpath
  Symlinking /builddir/venv3.11/lib64/python3.11/fnmatch.py
  Symlinking /builddir/venv3.11/lib64/python3.11/locale.py
  Symlinking /builddir/venv3.11/lib64/python3.11/encodings
  Cannot import bootstrap module: codecs
  Cannot import bootstrap module: stat
  Cannot import bootstrap module: UserDict
  File /builddir/venv3.11/lib64/python3.11/lib-dynload/readline.cpython-311-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: copy_reg
  Symlinking /builddir/venv3.11/lib64/python3.11/types.py
  Symlinking /builddir/venv3.11/lib64/python3.11/re
  Cannot import bootstrap module: sre
  Symlinking /builddir/venv3.11/lib64/python3.11/sre_parse.py
  Symlinking /builddir/venv3.11/lib64/python3.11/sre_constants.py
  Symlinking /builddir/venv3.11/lib64/python3.11/sre_compile.py
  File /builddir/venv3.11/lib64/python3.11/lib-dynload/zlib.cpython-311-x86_64-linux-gnu.so already exists
  Cannot import bootstrap module: _abcoll
  Symlinking /builddir/venv3.11/lib64/python3.11/warnings.py
  Symlinking /builddir/venv3.11/lib64/python3.11/linecache.py
  Cannot import bootstrap module: abc
  Cannot import bootstrap module: io
  Symlinking /builddir/venv3.11/lib64/python3.11/_weakrefset.py
  Symlinking /builddir/venv3.11/lib64/python3.11/copyreg.py
  Symlinking /builddir/venv3.11/lib64/python3.11/tempfile.py
  Symlinking /builddir/venv3.11/lib64/python3.11/random.py
  Symlinking /builddir/venv3.11/lib64/python3.11/__future__.py
  Symlinking /builddir/venv3.11/lib64/python3.11/collections
  Symlinking /builddir/venv3.11/lib64/python3.11/keyword.py
  Symlinking /builddir/venv3.11/lib64/python3.11/tarfile.py
  Symlinking /builddir/venv3.11/lib64/python3.11/shutil.py
  Symlinking /builddir/venv3.11/lib64/python3.11/struct.py
  Symlinking /builddir/venv3.11/lib64/python3.11/copy.py
  Symlinking /builddir/venv3.11/lib64/python3.11/tokenize.py
  Symlinking /builddir/venv3.11/lib64/python3.11/token.py
  Symlinking /builddir/venv3.11/lib64/python3.11/functools.py
  Symlinking /builddir/venv3.11/lib64/python3.11/heapq.py
  Symlinking /builddir/venv3.11/lib64/python3.11/bisect.py
  Symlinking /builddir/venv3.11/lib64/python3.11/weakref.py
  Symlinking /builddir/venv3.11/lib64/python3.11/reprlib.py
  Symlinking /builddir/venv3.11/lib64/python3.11/base64.py
  Cannot import bootstrap module: _dummy_thread
  Symlinking /builddir/venv3.11/lib64/python3.11/hashlib.py
  Symlinking /builddir/venv3.11/lib64/python3.11/hmac.py
  Symlinking /builddir/venv3.11/lib64/python3.11/imp.py
  Symlinking /builddir/venv3.11/lib64/python3.11/importlib
  Symlinking /builddir/venv3.11/lib64/python3.11/rlcompleter.py
  Symlinking /builddir/venv3.11/lib64/python3.11/operator.py
  Cannot import bootstrap module: _collections_abc
  Cannot import bootstrap module: _bootlocale
  Symlinking /builddir/venv3.11/lib64/python3.11/enum.py
Creating /builddir/venv3.11/lib/python3.11/site-packages
Writing /builddir/venv3.11/lib64/python3.11/site.py
Writing /builddir/venv3.11/lib64/python3.11/orig-prefix.txt
Writing /builddir/venv3.11/lib64/python3.11/no-global-site-packages.txt
Creating parent directories for /builddir/venv3.11/include
Symlinking /builddir/venv3.11/include/python3.11
Creating /builddir/venv3.11/bin
New python executable in /builddir/venv3.11/bin/python3.11
Changed mode of /builddir/venv3.11/bin/python3.11 to 0o755
Also creating executable in /builddir/venv3.11/bin/python
Changed mode of /builddir/venv3.11/bin/python to 0o755
Testing executable with /builddir/venv3.11/bin/python3.11 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
ERROR: The executable /builddir/venv3.11/bin/python3.11 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/builddir/venv3.11')
ERROR: virtualenv is not compatible with this system or executable



My best guess is that this code:

        filename = find_module_filename(modname)
        if filename is None:
            logger.info("Cannot import bootstrap module: %s" % modname)

in copy_required_modules() is chocking on https://docs.python.org/3.11/whatsnew/3.11.html#frozen-imports-static-code-objects


$ python3.6
...
>>> from virtualenv import find_module_filename
>>> find_module_filename('os')
'/usr/lib64/python3.6/os.py'

$ python3.10
...

>>> from virtualenv import find_module_filename
>>> find_module_filename('os')
'/usr/lib64/python3.10/os.py'

$ python3.11
...
>>> from virtualenv import find_module_filename
>>> find_module_filename('os')
(None)


I will stop for now and get back to this problem later.
Comment 6 Miro Hrončok 2023-02-01 18:12:06 UTC 
I've managed to solve that problem with:

        if not spec.has_location and spec.origin == "frozen":
            # https://docs.python.org/3.11/whatsnew/3.11.html#frozen-imports-static-code-objects
            # For frozen modules, we assume they are from /usr/lib64/python3.X/<name>.py
            import sysconfig  # always available on Python 3.11+ which has frozen modules
            return os.path.join(sysconfig.get_path("stdlib"), modname + ".py")


But now apparently venv3.11/bin/python isn't loading venv3.11/lib/python3.11/site.py when being executed and hence sys.path is missing the actual standard library.


$ venv3.6/bin/python -c 'import sys; print(sys.path)'
['', '/builddir/venv3.6/lib64/python36.zip', '/builddir/venv3.6/lib64/python3.6', '/builddir/venv3.6/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6', '/usr/lib/python3.6', '/builddir/venv3.6/lib/python3.6/site-packages']

$ venv3.11/bin/python -c 'import sys; print(sys.path)'
['', '/builddir/venv3.11/lib64/python311.zip', '/builddir/venv3.11/lib64/python3.11', '/builddir/venv3.11/lib64/python3.11/lib-dynload', '/builddir/venv3.11/lib64/python3.11/site-packages', '/builddir/venv3.11/lib/python3.11/site-packages']




Adding raise RuntimeError("bah") to venv3.X/lib/python3.X/site.py:

$ venv3.6/bin/python -c pass
Failed to import the site module
Traceback (most recent call last):
  File "/builddir/venv3.6/lib64/python3.6/site.py", line 66, in <module>
    raise RuntimeError("bah")
RuntimeError: bah

$ venv3.11/bin/python -c pass
(nothing)


That's... unfortunate.
Comment 7 Miro Hrončok 2023-02-02 11:10:38 UTC 
I have a working patch that does:

  $ virtualenv -p python3.11 venv3.11 
  Running virtualenv with interpreter /usr/bin/python3.11
  ERROR: Virtual environments created by virtualenv < 20 are not compatible with Python 3.11.
  ERROR: Use `python3.11 -m venv` instead.
  (exit code 1)

I believe this might be the most sensible approach this late in the RHEL 8 release cycle. Note that 3.12+ would need to be ported away from distutils anyway.
Comment 23 Petr Viktorin (pviktori) 2023-02-27 08:30:56 UTC 
> Please review the proposed note.

I'd prefer not calling virtualenv “legacy”. It's only legacy in RHEL -- it has a few more features, but not enough to justify Red Hat supporting it in the future.
Upstream, it's actively maintained.
I suggest dropping the word:

> .Creating virtual Python 3.11 environments fails when using the `virtualenv` utility

or maybe more positively:

> .Python 3.11 virtual environments should be created using `venv`, rather than the `virtualenv` utility

And the next sentence doesn't look correct. Virtual environments are specific to a version of Python. Pre-existing ones are not compatible with 3.11, but also with any version other that the one they were made for.
I suggest:

> The `virtualenv` utility in RHEL 8, provided by the `python3-virtualenv` package, is not compatible with Python 3.11. An attempt to use it to create a virtual environment will fail with the following error message:
Comment 24 Miro Hrončok 2023-02-27 08:49:32 UTC 
My understanding was that the "legacy" relates to the old version we ship, not to the tool itself. But it could be understood both ways and the adjustments by Petr make sense.
Comment 28 errata-xmlrpc 2023-05-16 08:43:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (python36:3.6 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2916
Note You need to log in before you can comment on or make changes to this bug.