RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1868003 - python36: Support for alternative architectures' names in Python importlib
Summary: python36: Support for alternative architectures' names in Python importlib
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: python3
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: beta
: 8.4
Assignee: Lumír Balhar
QA Contact: Lukáš Zachar
URL:
Whiteboard:
Depends On:
Blocks: 1868006
TreeView+ depends on / blocked
 
Reported: 2020-08-11 11:47 UTC by Lumír Balhar
Modified: 2021-05-18 14:57 UTC (History)
2 users (show)

Fixed In Version: python3-3.6.8-33.el8
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1868006 (view as bug list)
Environment:
Last Closed: 2021-05-18 14:56:51 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:1633 0 None None None 2021-05-18 14:57:28 UTC

Description Lumír Balhar 2020-08-11 11:47:52 UTC
I have recently discovered that our Pythons in Fedora and RHEL are configured in a different way than upstream and some other distributions. The difference is in names for some architectures — for example, for 64 bit PowerPC, Fedora and RHEL use "ppc64" but CPython upstream, Debian, SUSE, etc. use "powerpc64".

This difference causes trouble with universal manylinux wheels. For example, simple-manylinux-demo produces a wheel for PowerPC with a built extension module. The module has powerpc64le in its name and therefore Python interpreters on RHEL cannot find it and import it.

The plan is to patch all Python interpreters in RHEL to be compatible with the upstream and downstream names at the same time and then we will remove the patch causing the difference.

This is already fixed in Fedora, for example https://src.fedoraproject.org/rpms/python3.9/c/898f7432fb5857a33df7360d48aff9f8fcba03f8?branch=master

Further reading: https://github.com/pypa/manylinux/issues/687

Comment 1 Petr Viktorin (pviktori) 2020-08-12 12:21:56 UTC
To test this, install and run the third-party Numpy package on ppc64le. The test currently exists (I'm not sure where) but is failing.

Comment 2 Lumír Balhar 2020-08-13 07:26:01 UTC
This can wait for RHEL 8.4.0 because it needs some time in Fedora first.

The change is ready in https://src.osci.redhat.com/fork/lbalhar/rpms/python3/c/9cd247471bd589025c08ae66351123499b6b21bd?branch=rhel-8.4.0
Scratch build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=30731618

Verification:

old:

# python3.6 -c "import sys; print(sys.path_importer_cache['/usr/lib64/python3.6']._loaders)"
[('.cpython-36m-ppc64le-linux-gnu.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.abi3.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.py', <class '_frozen_importlib_external.SourceFileLoader'>), ('.pyc', <class '_frozen_importlib_external.SourcelessFileLoader'>)]

# python3.6 -m unittest discover dummyextension.tests
E
======================================================================
ERROR: dummyextension.tests.test_extension (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: dummyextension.tests.test_extension
Traceback (most recent call last):
  File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib64/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/usr/local/lib64/python3.6/site-packages/dummyextension/tests/test_extension.py", line 2, in <module>
    from dummyextension.extension import hello
ModuleNotFoundError: No module named 'dummyextension.extension'


----------------------------------------------------------------------
Ran 1 test in 0.000s

new:

# python3.6 -c "import sys; print(sys.path_importer_cache['/usr/lib64/python3.6']._loaders)"
[('.cpython-36m-ppc64le-linux-gnu.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.abi3.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.cpython-36m-powerpc64le-linux-gnu.so', <class '_frozen_importlib_external.ExtensionFileLoader'>), ('.py', <class '_frozen_importlib_external.SourceFileLoader'>), ('.pyc', <class '_frozen_importlib_external.SourcelessFileLoader'>)]
# python3.6 -m unittest discover dummyextension.tests
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Comment 5 Lumír Balhar 2020-11-03 10:07:41 UTC
PR: https://src.osci.redhat.com/rpms/python3/pull-request/63

Verification:

Upstream manylinux wheels do not work with the older version:

(platform-python-3.6.8-31.el8.ppc64le)

# python3 -m venv venv
# source venv/bin/activate
(venv) # pip install -U pip
(venv) # pip install simple-manylinux-demo
Collecting simple-manylinux-demo
  Downloading simple_manylinux_demo-1.0-cp36-cp36m-manylinux2014_ppc64le.whl (17 kB)
Installing collected packages: simple-manylinux-demo
Successfully installed simple-manylinux-demo-1.0
(venv) # ls -la venv/lib/python3.6/site-packages/dummyextension/
total 120
drwxr-xr-x.  4 root root  4096 Nov  3 09:36 .
drwxr-xr-x. 10 root root  4096 Nov  3 09:36 ..
-rw-r--r--.  1 root root     0 Nov  3 09:36 __init__.py
drwxr-xr-x.  2 root root  4096 Nov  3 09:36 __pycache__
-rwxr-xr-x.  1 root root 89304 Nov  3 09:36 extension.cpython-36m-powerpc64le-linux-gnu.so
drwxr-xr-x.  3 root root  4096 Nov  3 09:36 tests
(venv) # python
Python 3.6.8 (default, Aug 18 2020, 12:45:08) 
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dummyextension import extension
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'extension'
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-36m-ppc64le-linux-gnu.so'

The upstream name works in the new version (platform-python-3.6.8-32.el8.ppc64le) while it keeps compatibility with the old downstream name and generates own extensions with it:

# python3 -m venv venv
# source venv/bin/activate
(venv) # pip install -U pip
(venv) # pip install simple-manylinux-demo
(venv) # python
Python 3.6.8 (default, Nov  3 2020, 02:08:33) 
[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dummyextension import extension
>>> 
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-36m-ppc64le-linux-gnu.so'
>>> 

(venv) # git clone https://github.com/frenzymadness/simple-manylinux-demo.git
(venv) # cd simple-manylinux-demo/
(venv) # python setup.py build
(venv) # ls -la build
total 32
drwxr-xr-x. 4 root root 4096 Nov  3 09:45 .
drwxr-xr-x. 5 root root 4096 Nov  3 09:45 ..
drwxr-xr-x. 3 root root 4096 Nov  3 09:45 lib.linux-ppc64le-3.6
drwxr-xr-x. 3 root root 4096 Nov  3 09:45 temp.linux-ppc64le-3.6
(venv) # ls -la build/lib.linux-ppc64le-3.6/dummyextension/
total 56
drwxr-xr-x. 3 root root  4096 Nov  3 09:47 .
drwxr-xr-x. 3 root root  4096 Nov  3 09:45 ..
-rw-r--r--. 1 root root     0 Nov  3 09:45 __init__.py
-rwxr-xr-x. 1 root root 89544 Nov  3 09:47 extension.cpython-36m-ppc64le-linux-gnu.so
drwxr-xr-x. 2 root root  4096 Nov  3 09:45 tests
(venv) # PYTHONPATH=build/lib.linux-ppc64le-3.6/dummyextension/ python
Python 3.6.8 (default, Nov  3 2020, 02:08:33) 
[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import extension
>>>

Comment 6 Lumír Balhar 2020-11-25 07:26:42 UTC
Verified manually one more time after the main build. Build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=33244250

Comment 7 Lumír Balhar 2020-12-01 12:11:24 UTC
Gating is done although I had to wave one broken test.

Let's wait for the new automation which should add this bug and build to automatic erratum.

Comment 13 errata-xmlrpc 2021-05-18 14:56:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: python3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1633


Note You need to log in before you can comment on or make changes to this bug.