Bug 2176158 - python-googleapis-common-protos fails to build with Python 3.12: TypeError: Metaclasses with custom tp_new are not supported.
Summary: python-googleapis-common-protos fails to build with Python 3.12: TypeError: M...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: protobuf
Version: 39
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Major Hayden 🤠
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.12
TreeView+ depends on / blocked
 
Reported: 2023-03-07 14:30 UTC by Tomáš Hrnčiar
Modified: 2023-10-12 12:57 UTC (History)
13 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2023-10-12 12:57:59 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github protocolbuffers protobuf issues 12186 0 None open Compiled Python extension does not work with Python 3.12 2023-03-09 14:55:58 UTC
Github protocolbuffers protobuf issues 12187 0 None open Pure-Python extension does not work with Python 3.12 2023-03-09 15:59:04 UTC

Description Tomáš Hrnčiar 2023-03-07 14:30:40 UTC
python-googleapis-common-protos fails to build with Python 3.12.0a5.

Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 171, in <module>
    main()
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 167, in main
    import_modules(modules)
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 100, in import_modules
    importlib.import_module(module)
  File "/usr/lib64/python3.12/importlib/__init__.py", line 124, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1334, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1307, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1278, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 841, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1012, in exec_module
  File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
  File "/builddir/build/BUILDROOT/python-googleapis-common-protos-1.58.0-2.fc39~bootstrap.x86_64/usr/lib/python3.12/site-packages/google/api/annotations_pb2.py", line 20, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/usr/lib64/python3.12/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
TypeError: Metaclasses with custom tp_new are not supported.

Added the new limited C API function PyType_FromMetaclass(), which generalizes the existing PyType_FromModuleAndSpec() using an additional metaclass argument. (Contributed by Wenzel Jakob in gh-93012.)

The metaclass is used to construct the resulting type object. When metaclass is NULL, the metaclass is derived from bases (or Py_tp_base[s] slots if bases is NULL, see below). Note that metaclasses that override tp_new are not supported.

https://github.com/python/cpython/issues/93012
https://docs.python.org/3.12/whatsnew/3.12.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.12/fedora-rawhide-x86_64/05572573-python-googleapis-common-protos/

For all our attempts to build python-googleapis-common-protos with Python 3.12, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/package/python-googleapis-common-protos/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/

Let us know here if you have any questions.

Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Ben Beasley 2023-03-09 14:55:58 UTC
The problem here is actually in the Python bindings for protobuf.

It can probably be worked around by disabling the C++ version of the Python bindings on Python 3.12, at the cost of a significant performance regression: https://src.fedoraproject.org/rpms/protobuf/pull-request/19

However, that introduces a dependency on python3dist(six), which isn’t yet available in the Python 3.12 COPR, so I can’t test it yet.

I have reported the issue upstream: https://github.com/protocolbuffers/protobuf/issues/12186

Comment 2 Ben Beasley 2023-03-09 14:58:42 UTC
It actually does look like python-six has built successfully in the COPR. I’m not sure why my local mock build using the COPR repos disagrees, even after removing all caches. Let’s see what the COPR build for the PR does: https://copr.fedorainfracloud.org/coprs/g/python/python3.12/build/5618884/

Comment 3 Ben Beasley 2023-03-09 15:53:32 UTC
OK, with `--no-bootstrap-chroot` I am able to build https://src.fedoraproject.org/rpms/protobuf/pull-request/19 locally in `mock`. It does fix the reported traceback (metaclass issue), but there is still an incompatibility in the protobuf code.

Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 171, in <module>
    main()
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 167, in main
    import_modules(modules)
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 100, in import_modules
    importlib.import_module(module)
  File "/usr/lib64/python3.12/importlib/__init__.py", line 124, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1334, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1307, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1278, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 841, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1015, in exec_module
  File "<frozen importlib._bootstrap>", line 400, in _call_with_frames_removed
  File "/builddir/build/BUILDROOT/python-googleapis-common-protos-1.58.0-1.fc39~bootstrap.x86_64/usr/lib/python3.12/site-packages/google/api/annotations_pb2.py", line 31, in <module>
    from google.api import http_pb2 as google_dot_api_dot_http__pb2
  File "/builddir/build/BUILDROOT/python-googleapis-common-protos-1.58.0-1.fc39~bootstrap.x86_64/usr/lib/python3.12/site-packages/google/api/http_pb2.py", line 31, in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/google/protobuf/descriptor_pool.py", line 216, in AddSerializedFile
    from google.protobuf import descriptor_pb2  
  File "/usr/lib/python3.12/site-packages/google/protobuf/descriptor_pb2.py", line 1883, in <module>
    FileDescriptorSet = _reflection.GeneratedProtocolMessageType('FileDescriptorSet', (_message.Message,), {
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/google/protobuf/internal/python_message.py", line 198, in __init__
    _AddPropertiesForFields(descriptor, cls)
  File "/usr/lib/python3.12/site-packages/google/protobuf/internal/python_message.py", line 586, in _AddPropertiesForFields
    _AddPropertiesForField(field, cls)
  File "/usr/lib/python3.12/site-packages/google/protobuf/internal/python_message.py", line 612, in _AddPropertiesForField
    _AddPropertiesForRepeatedField(field, cls)  
  File "/usr/lib/python3.12/site-packages/google/protobuf/internal/python_message.py", line 668, in _AddPropertiesForRepeatedField
    setattr(cls, property_name, _FieldProperty(field, getter, setter, doc=doc))
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/google/protobuf/internal/python_message.py", line 623, in __init__
    property.__init__(self, getter, setter, doc=doc)
AttributeError: '_FieldProperty' object attribute '__doc__' is read-only

Comment 5 Major Hayden 🤠 2023-05-17 13:41:56 UTC
Thank you for chasing this down, Ben!

Comment 6 Fedora Release Engineering 2023-08-16 07:11:04 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle.
Changing version to 39.

Comment 7 Ben Beasley 2023-10-12 12:57:59 UTC
This was fixed in Python itself prior to the 3.12 final release.

https://github.com/protocolbuffers/protobuf/issues/12187#issuecomment-1579476322


Note You need to log in before you can comment on or make changes to this bug.