Bug 2255342

Summary:	python-twisted fails to build with Python 3.13: ModuleNotFoundError: No module named 'cgi' & many other failures
Product:	[Fedora] Fedora	Reporter:	Karolina Surma <ksurma>
Component:	python-twisted	Assignee:	Jonathan Steffan <jonathansteffan>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	eclipseo, jonathan, jonathansteffan, ksurma, mhroncok, python-packagers-sig
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2025-01-17 16:42:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2316869
Bug Blocks:	2244836

Description Karolina Surma 2023-12-20 08:50:21 UTC

python-twisted fails to build with Python 3.13.0a2.

Test suite result:
FAILED (skips=640, failures=14, errors=51, successes=9085)

See the build logs for the full results:

https://copr-be.cloud.fedoraproject.org/results/@python/python3.13/fedora-rawhide-x86_64/06770350-python-twisted/

According to https://docs.python.org/3.13/whatsnew/3.13.html module cgi was removed:

PEP 594: Remove the cgi and cgitb modules, deprecated in Python 3.11.

cgi.FieldStorage can typically be replaced with urllib.parse.parse_qsl() for GET and HEAD requests, and the email.message module or multipart PyPI project for POST and PUT.

cgi.parse() can be replaced by calling urllib.parse.parse_qs() directly on the desired query string, except for multipart/form-data input, which can be handled as described for cgi.parse_multipart().

cgi.parse_multipart() can be replaced with the functionality in the email package (e.g. email.message.EmailMessage and email.message.Message) which implements the same MIME RFCs, or with the multipart PyPI project.

cgi.parse_header() can be replaced with the functionality in the email package, which implements the same MIME RFCs. For example, with email.message.EmailMessage:

from email.message import EmailMessage
msg = EmailMessage()
msg['content-type'] = 'application/json; charset="utf8"'
main, params = msg.get_content_type(), msg['content-type'].params

(Contributed by Victor Stinner in gh-104773.)


https://docs.python.org/3.13/whatsnew/3.13.html

For all our attempts to build python-twisted with Python 3.13, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.13/package/python-twisted/

Testing and mass rebuild of packages is happening in copr.
You can follow these instructions to test locally in mock if your package builds with Python 3.13:
https://copr.fedorainfracloud.org/coprs/g/python/python3.13/

Let us know here if you have any questions.

Python 3.13 is planned to be included in Fedora 41.
To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.13.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires),
so if this package is required a lot, it's important for us to get it fixed soon.

We'd appreciate help from the people who know this package best,
but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2023-12-20 09:36:28 UTC

This is also affected by https://github.com/twisted/twisted/issues/12052 which also impacts builds with Python 3.12.1

https://koschei.fedoraproject.org/package/python-twisted shows
FAILED (skips=689, failures=2, errors=8, successes=10369)

Comment 2 Miro Hrončok 2023-12-20 12:03:48 UTC

(In reply to Miro Hrončok from comment #1)
> This is also affected by https://github.com/twisted/twisted/issues/12052
> which also impacts builds with Python 3.12.1
> 
> https://koschei.fedoraproject.org/package/python-twisted shows
> FAILED (skips=689, failures=2, errors=8, successes=10369)

This part should be fixed via https://src.fedoraproject.org/rpms/python-twisted/pull-request/22

Comment 3 Miro Hrončok 2023-12-20 12:13:57 UTC

With https://src.fedoraproject.org/rpms/python-twisted/pull-request/22 I get this with Python 3.13:

FAILED (skips=640, failures=12, errors=49, successes=9087)

29 of this is builtins.ModuleNotFoundError: No module named 'cgi' -> should be fixed by https://github.com/twisted/twisted/pull/12058

Comment 4 Karolina Surma 2024-01-16 10:16:21 UTC

With cgi fixes backported in https://src.fedoraproject.org/rpms/python-twisted/pull-request/23 it's better, but not completely:

FAILED (skips=709, failures=13, errors=20, successes=10324)

Errors come from:

[ERROR]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/web/test/test_html.py", line 32, in test_deprecation
    assertDeprecationWarningOf("PRE")
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/web/test/test_html.py", line 22, in assertDeprecationWarningOf
    warningsShown = self.flushWarnings([self.test_deprecation])
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/trial/_synctest.py", line 1196, in flushWarnings
    if not (min(lineNumbers) <= aWarning.lineno <= max(lineNumbers)):
builtins.TypeError: '<' not supported between instances of 'int' and 'NoneType'

twisted.web.test.test_html.WebHtmlTests.test_deprecation

Failures are more diverse, but 5 of them comes from the stripped whitespace characters.

Comment 5 Karolina Surma 2024-01-16 13:58:50 UTC

With another added backport in the above linked PR, we're down to 10 failures. The non-whitespace-related ones are below:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/test/test_inlinecb.py", line 192, in test_forwardLotsOfTracebacks
    self.assertIn("in erroring", tb)
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/trial/_synctest.py", line 509, in assertIn
    raise self.failureException(msg or f"{containee!r} not in {container!r}")
twisted.trial.unittest.FailTest: 'in erroring' not in 'Traceback (most recent call last):\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/python/util.py", line 958, in runWithWarningsSuppressed\n    return f(*args, **kwargs)\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/test/test_inlinecb.py", line 189, in test_forwardLotsOfTracebacks\n    d = calling()\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/defer.py", line 2256, in unwindGenerator\n    return _cancellableInlineCallbacks(gen)\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/defer.py", line 2168, in _cancellableInlineCallbacks\n    _inlineCallbacks(None, gen, status, _copy_context())\n--- <exception caught here> ---\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/defer.py", line 1996, in _inlineCallbacks\n    result = context.run(\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator\n    return g.throw(self.value.with_traceback(self.tb))\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/test/test_inlinecb.py", line 187, in calling\n    yield calling2()\nbuiltins.Exception: Error Marker\n'

twisted.internet.test.test_inlinecb.ForwardTraceBackTests.test_forwardLotsOfTracebacks
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/test/test_inlinecb.py", line 149, in test_forwardTracebacks
    self.assertIn("in erroring", tb)
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/trial/_synctest.py", line 509, in assertIn
    raise self.failureException(msg or f"{containee!r} not in {container!r}")
twisted.trial.unittest.FailTest: 'in erroring' not in 'Traceback (most recent call last):\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/python/util.py", line 958, in runWithWarningsSuppressed\n    return f(*args, **kwargs)\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/test/test_inlinecb.py", line 146, in test_forwardTracebacks\n    d = calling()\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/defer.py", line 2256, in unwindGenerator\n    return _cancellableInlineCallbacks(gen)\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/defer.py", line 2168, in _cancellableInlineCallbacks\n    _inlineCallbacks(None, gen, status, _copy_context())\n--- <exception caught here> ---\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/defer.py", line 1996, in _inlineCallbacks\n    result = context.run(\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator\n    return g.throw(self.value.with_traceback(self.tb))\n  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/internet/test/test_inlinecb.py", line 144, in calling\n    yield erroring()\nbuiltins.Exception: Error Marker\n'

twisted.internet.test.test_inlinecb.ForwardTraceBackTests.test_forwardTracebacks
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/python/test/test_shellcomp.py", line 362, in test_poorlyDescribedOptMethod
    self.assertEqual(descr, "silly")
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/trial/_synctest.py", line 444, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib64/python3.13/unittest/case.py", line 887, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.13/unittest/case.py", line 1253, in assertMultiLineEqual
    self.fail(self._formatMessage(msg, standardMsg))
twisted.trial.unittest.FailTest: '' != 'silly'
+ silly


twisted.python.test.test_shellcomp.ZshTests.test_poorlyDescribedOptMethod
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/test/test_failure.py", line 967, in test_findFailureInGenerator
    self.assertEqual(foundFailures, [f])
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/trial/_synctest.py", line 444, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib64/python3.13/unittest/case.py", line 887, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.13/unittest/case.py", line 1093, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/usr/lib64/python3.13/unittest/case.py", line 1075, in assertSequenceEqual
    self.fail(msg)
twisted.trial.unittest.FailTest: Lists differ: [None] != [<twisted.python.failure.Failure builtins.ZeroDivisionError: division by zero>]

First differing element 0:
None
<twisted.python.failure.Failure builtins.ZeroDivisionError: division by zero>

- [None]
+ [<twisted.python.failure.Failure builtins.ZeroDivisionError: division by zero>]

twisted.test.test_failure.ExtendedGeneratorTests.test_findFailureInGenerator
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/web/test/test_cgi.py", line 389, in _test_urlParameters_1
    self.assertEqual(res, expected)
  File "/builddir/build/BUILD/twisted-twisted-23.10.0/src/twisted/trial/_synctest.py", line 444, in assertEqual
    super().assertEqual(first, second, msg)
  File "/usr/lib64/python3.13/unittest/case.py", line 887, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.13/unittest/case.py", line 880, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: b'\n<html>\n  <head><title>500 - CGI Script[125 chars]l>\n' != b'1234\n'

twisted.web.test.test_cgi.CGIScriptTests.test_urlParameters

Comment 6 Karolina Surma 2024-01-30 14:28:27 UTC

With some more upstream fixes included into the package, three tests fail: twisted.test.test_failure.ExtendedGeneratorTests.test_findFailureInGenerator, twisted.web.test.test_cgi.CGIScriptTests.test_urlParameters and test_flatten (related to the changed traceback output). Reported upstream.

Comment 7 Aoife Moloney 2024-02-15 23:08:11 UTC

This bug appears to have been reported against 'rawhide' during the Fedora Linux 40 development cycle.
Changing version to 40.