Bug 1890878

Summary: libxml2 fails to build with Python 3.10: SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
Product: [Fedora] Fedora Reporter: Tomáš Hrnčiar <thrnciar>
Component: libxml2Assignee: Igor Raits <igor.raits>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: fedora, igor.raits, ljavorsk, mhroncok, thrnciar, veillard, vstinner
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libxml2-2.9.10-9.fc34 libxml2-2.9.10-9.eln105 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-29 12:54:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1890881    

Description Tomáš Hrnčiar 2020-10-23 07:49:36 UTC
libxml2 fails to build with Python 3.10.0a1.

make[2]: Leaving directory '/builddir/build/BUILD/libxml2-2.9.10/py3/python/tests'
test.xml:1: parser error : Couldn't find end of Start Tag foo
<foo
    ^
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/builddir/build/BUILD/libxml2-2.9.10/py3/python/tests/../../../python/tests/push.py", line 9, in <module>
    ctxt.parseChunk("/>", 2, 1)
  File "/builddir/build/BUILD/libxml2-2.9.10/python/libxml2.py", line 5101, in parseChunk
    ret = libxml2mod.xmlParseChunk(self._o, chunk, size, terminate)
SystemError: <built-in function xmlParseChunk> returned a result with an error set

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01715877-libxml2/

For all our attempts to build libxml2 with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/libxml2/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2020-10-23 08:18:10 UTC
https://docs.python.org/3.10/whatsnew/3.10.html#id2

The PY_SSIZE_T_CLEAN macro must now be defined to use PyArg_ParseTuple() and Py_BuildValue() formats which use #: es#, et#, s#, u#, y#, z#, U# and Z#. See Parsing arguments and building values and the PEP 353.

https://docs.python.org/3.10/c-api/arg.html#arg-parsing
https://www.python.org/dev/peps/pep-0353
https://bugs.python.org/issue40943

Comment 2 Victor Stinner 2020-11-09 17:26:11 UTC
I reported the issue to upstream: https://gitlab.gnome.org/GNOME/libxml2/-/issues/203

And I proposed a fix: https://gitlab.gnome.org/GNOME/libxml2/-/merge_requests/87

Comment 3 Victor Stinner 2020-11-10 14:02:01 UTC
I proposed to https://src.fedoraproject.org/rpms/libxml2/pull-request/10 to backport my fix and fix the package (always regenerate the 3 Python generated files).

Comment 5 Fedora Update System 2020-11-12 11:54:15 UTC
FEDORA-2020-576ed43594 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 6 Fedora Update System 2020-11-12 12:09:16 UTC
FEDORA-2020-160ee73cea has been pushed to the Fedora ELN stable repository.
If problem still persists, please make note of it in this bug report.

Comment 7 Victor Stinner 2020-11-19 20:17:22 UTC
> I proposed to https://src.fedoraproject.org/rpms/libxml2/pull-request/10 to backport my fix and fix the package (always regenerate the 3 Python generated files).

FYI my PR has been merged upstream:
* https://gitlab.gnome.org/GNOME/libxml2/-/commit/f42a0524c693a6d52e95a37c9cf04df1be48262c
* https://gitlab.gnome.org/GNOME/libxml2/-/commit/ac5e99911a45ad6f57a6aa7b33fd2de9da72aa72

Comment 8 Lukas Javorsky 2021-09-29 10:23:45 UTC
Hi, I've encountered this issue on Fedora 35 container.

This issue occurs when I try to build the module build for RHEL-8 using this command:
rhpkg module-build --scratch --buildrequires platform:el8.5.0

The Fedora container that I'm using is here:
https://gitlab.cee.redhat.com/platform-eng-core-services/core-services-dockerfiles/-/tree/master/pkg-container

Error message: Could not execute module_build: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

PS: I've tried it in the Fedora 34 container and it worked as expected, so the issue must be in the commits between these two versions

Reopening this BZ, if you prefer to create a new one let me know.

Comment 9 Miro Hrončok 2021-09-29 10:36:25 UTC
I am not sure that gitlab.cee link works outside of the Red Hat network and this is  Fedora Bugzilla. Could you provide a more accessible reproducer, please?


Also, what makes you think that the error comes from libxml2?

Comment 10 Miro Hrončok 2021-09-29 10:39:04 UTC
A list of installed packages (including their versions, releaes and architecture) would be helpful to determine what packages could cause this. Could you please attach the output of `rpm -qa`?

Also, is there a way to see the full traceback?

Comment 12 Miro Hrončok 2021-09-29 12:46:10 UTC
> Also, is there a way to see the full traceback?

Please, run again with rhpkg --debug. That raises the exception instead of hiding it:

    except Exception as e:
        ...

        if getattr(client.args, 'debug', False):
            raise
        log.error('Could not execute %s: %s' % (client.args.command.__name__, e))
        sys.exit(1)

Comment 13 Miro Hrončok 2021-09-29 12:54:36 UTC
I've got it:

Traceback (most recent call last):
  File "/usr/bin/rhpkg", line 16, in <module>
    main()
  File "/usr/lib/python3.10/site-packages/rhpkg/__main__.py", line 104, in main
    sys.exit(client.args.command())
  File "/usr/lib/python3.10/site-packages/pyrpkg/cli.py", line 2296, in module_build
    build_ids = self._cmd.module_submit_build(
  File "/usr/lib/python3.10/site-packages/pyrpkg/__init__.py", line 3863, in module_submit_build
    resp = self.module_send_authorized_request(
  File "/usr/lib/python3.10/site-packages/pyrpkg/__init__.py", line 3786, in module_send_authorized_request
    resp = requests.request(verb, url, data=data, auth=auth, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 662, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests_kerberos/kerberos_.py", line 385, in handle_response
    self.cbt_struct = kerberos.channelBindings(application_data=cbt_application_data)
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats



A reproducer:

>>> import kerberos
>>> kerberos.__file__
'/usr/lib64/python3.10/site-packages/kerberos.cpython-310-x86_64-linux-gnu.so'
>>> kerberos.channelBindings(application_data={})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

Comment 15 Miro Hrončok 2021-09-29 13:17:10 UTC
Reported as bz2008899.