python-aiohttp failed to build from source in Fedora rawhide/f44 https://koji.fedoraproject.org/koji/taskinfo?taskID=141190754 For details on the mass rebuild see: https://fedoraproject.org/wiki/Fedora_44_Mass_Rebuild Please fix python-aiohttp at your earliest convenience and set the bug's status to ASSIGNED when you start fixing it. If the bug remains in NEW state for 8 weeks, python-aiohttp will be orphaned. Before branching of Fedora 45, python-aiohttp will be retired, if it still fails to build. For more details on the FTBFS policy, please visit: https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
Created attachment 2126594 [details] build.log file build.log too big, will only attach last 32768 bytes
Created attachment 2126595 [details] root.log file root.log too big, will only attach last 32768 bytes
Created attachment 2126596 [details] state.log
This is a s390x-only test failure: =================================== FAILURES =================================== ___________________ test_send_compress_text_notakeover[zlib] ___________________ [gw1] linux -- Python 3.14.2 /usr/bin/python3 self = <Mock name='mock.write' id='4394706665552'> args = (b'\xc1\x06*I\xad(\x01\x00',), kwargs = {} expected = call(b'\xc1\x06*I\xad(\x01\x00') actual = call(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xe...xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') _error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x3ff38e7bc10> cause = None def assert_called_with(self, /, *args, **kwargs): """assert that the last call was made with the specified arguments. Raises an AssertionError if the args and keyword args passed in are different to the last call to the mock.""" if self.call_args is None: expected = self._format_mock_call_signature(args, kwargs) actual = 'not called.' error_message = ('expected call not found.\nExpected: %s\n Actual: %s' % (expected, actual)) raise AssertionError(error_message) def _error_message(): msg = self._format_mock_failure_message(args, kwargs) return msg expected = self._call_matcher(_Call((args, kwargs), two=True)) actual = self._call_matcher(self.call_args) if actual != expected: cause = expected if isinstance(expected, Exception) else None > raise AssertionError(_error_message()) from cause E AssertionError: expected call not found. E Expected: write(b'\xc1\x06*I\xad(\x01\x00') E Actual: write(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') _error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x3ff38e7bc10> actual = call(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xe...xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') args = (b'\xc1\x06*I\xad(\x01\x00',) cause = None expected = call(b'\xc1\x06*I\xad(\x01\x00') kwargs = {} self = <Mock name='mock.write' id='4394706665552'> /usr/lib64/python3.14/unittest/mock.py:985: AssertionError During handling of the above exception, another exception occurred: protocol = <Mock id='4394705893904'>, transport = <Mock id='4394705894576'> @pytest.mark.usefixtures("parametrize_zlib_backend") async def test_send_compress_text_notakeover(protocol, transport) -> None: compress_obj = ZLibBackend.compressobj(level=ZLibBackend.Z_BEST_SPEED, wbits=-15) writer = WebSocketWriter(protocol, transport, compress=15, notakeover=True) msg = ( compress_obj.compress(b"text") + compress_obj.flush(ZLibBackend.Z_FULL_FLUSH) ).removesuffix(b"\x00\x00\xff\xff") await writer.send_frame(b"text", WSMsgType.TEXT) writer.transport.write.assert_called_with( # type: ignore[attr-defined] b"\xc1" + len(msg).to_bytes(1, "big") + msg ) await writer.send_frame(b"text", WSMsgType.TEXT) > writer.transport.write.assert_called_with( # type: ignore[attr-defined] b"\xc1" + len(msg).to_bytes(1, "big") + msg ) E AssertionError: expected call not found. E Expected: write(b'\xc1\x06*I\xad(\x01\x00') E Actual: write(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') E E pytest introspection follows: E E Args: compress_obj = <zlib.Compress object at 0x3ff64318340> msg = b'*I\xad(\x01\x00' protocol = <Mock id='4394705893904'> transport = <Mock id='4394705894576'> writer = <aiohttp._websocket.writer.WebSocketWriter object at 0x3ff38ea0d70> tests/test_websocket_writer.py:122: AssertionError _____________ test_send_compress_text_notakeover[zlib_ng.zlib_ng] ______________ [gw1] linux -- Python 3.14.2 /usr/bin/python3 self = <Mock name='mock.write' id='4394706671264'> args = (b'\xc1\x06*I\xad(\x01\x00',), kwargs = {} expected = call(b'\xc1\x06*I\xad(\x01\x00') actual = call(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xe...xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') _error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x3ff38e7aa30> cause = None def assert_called_with(self, /, *args, **kwargs): """assert that the last call was made with the specified arguments. Raises an AssertionError if the args and keyword args passed in are different to the last call to the mock.""" if self.call_args is None: expected = self._format_mock_call_signature(args, kwargs) actual = 'not called.' error_message = ('expected call not found.\nExpected: %s\n Actual: %s' % (expected, actual)) raise AssertionError(error_message) def _error_message(): msg = self._format_mock_failure_message(args, kwargs) return msg expected = self._call_matcher(_Call((args, kwargs), two=True)) actual = self._call_matcher(self.call_args) if actual != expected: cause = expected if isinstance(expected, Exception) else None > raise AssertionError(_error_message()) from cause E AssertionError: expected call not found. E Expected: write(b'\xc1\x06*I\xad(\x01\x00') E Actual: write(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') _error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x3ff38e7aa30> actual = call(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xe...xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') args = (b'\xc1\x06*I\xad(\x01\x00',) cause = None expected = call(b'\xc1\x06*I\xad(\x01\x00') kwargs = {} self = <Mock name='mock.write' id='4394706671264'> /usr/lib64/python3.14/unittest/mock.py:985: AssertionError During handling of the above exception, another exception occurred: protocol = <Mock id='4394706669584'>, transport = <Mock id='4394706670592'> @pytest.mark.usefixtures("parametrize_zlib_backend") async def test_send_compress_text_notakeover(protocol, transport) -> None: compress_obj = ZLibBackend.compressobj(level=ZLibBackend.Z_BEST_SPEED, wbits=-15) writer = WebSocketWriter(protocol, transport, compress=15, notakeover=True) msg = ( compress_obj.compress(b"text") + compress_obj.flush(ZLibBackend.Z_FULL_FLUSH) ).removesuffix(b"\x00\x00\xff\xff") await writer.send_frame(b"text", WSMsgType.TEXT) writer.transport.write.assert_called_with( # type: ignore[attr-defined] b"\xc1" + len(msg).to_bytes(1, "big") + msg ) await writer.send_frame(b"text", WSMsgType.TEXT) > writer.transport.write.assert_called_with( # type: ignore[attr-defined] b"\xc1" + len(msg).to_bytes(1, "big") + msg ) E AssertionError: expected call not found. E Expected: write(b'\xc1\x06*I\xad(\x01\x00') E Actual: write(b'\xc1G\xec\xfd]w\x1bG\xb6\xad\xf9\xeb\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\xfd\x95\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef\xef/\xf1\xfe\xfe\xfe\xfe\xb2\x16\x07%\x0f\xc7\x08\x00') E E pytest introspection follows: E E Args: compress_obj = <zlib_ng._Compress object at 0x3ff38ea1390> msg = b'*I\xad(\x01\x00' protocol = <Mock id='4394706669584'> transport = <Mock id='4394706670592'> writer = <aiohttp._websocket.writer.WebSocketWriter object at 0x3ff38ea1130> tests/test_websocket_writer.py:122: AssertionError This used to work. According to https://koji.fedoraproject.org/koji/packageinfo?packageID=19224, the last successful build was on 2025-10-17, so this broke sometime after then. Unfortunately, that’s all we have to go on since Koschei doesn’t do scratch builds for s390x.
It looks like this may be checking for a precise compressed-data byte string, which isn’t necessarily something one can expect across various zlib implementations. The weird thing, which perhaps looks like a bug, is that the actual message is much larger than the expected one. I want to check if this regression is connected to https://src.fedoraproject.org/rpms/zlib-ng/c/c93c81c63053230109a4d07792e9e82ea706af59?branch=rawhide or not.
Hi Ben, are you looking into this or should I? I believe there is an environment variable that force zlib-ng to the generic algorithms for cases like this. I am pretty sure saw that recently used in some package.
(In reply to Dan Horák from comment #6) > Hi Ben, are you looking into this or should I? I believe there is an > environment variable that force zlib-ng to the generic algorithms for cases > like this. I am pretty sure saw that recently used in some package. Thanks. If you’re able to investigate, that would be helpful, especially since you can comfortably test on real hardware. I couldn’t reproduce this in a qemu-user-static emulated fedora-rawhide-s390x chroot, which would seem to support the theory that the DFLTCC code paths are producing possibly-valid but significantly longer than necessary streams, at least for some short payloads.
So it is the new zlib-ng build with the DFLTCC feature enabled. Downgrading to zlib-ng-2.3.2-1.fc44 makes the failures go away. Now to figure out how to avoid using the accelerated routines ... Tulio, any chance you know the trick ^^ ?
I don't think that's possible. The PR adding the environment variable was not accepted upstream. Ben, do you have the length of the actual compressed data?
(In reply to Tulio Magno Quites Machado Filho from comment #9) > I don't think that's possible. The PR adding the environment variable was > not accepted upstream. > > Ben, do you have the length of the actual compressed data? Hmm, https://bugzilla.redhat.com/show_bug.cgi?id=2434949#c4 shows byte strings in the output, but across https://github.com/aio-libs/aiohttp/blob/0cba7986c4764d32c062e7ea43155a289893e4f9/tests/test_websocket_writer.py#L116-L133 and all the indirection in https://github.com/aio-libs/aiohttp/blob/master/aiohttp/compression_utils.py there’s quite a bit of complexity in what’s actually happening. I briefly tried to craft a trivial demonstration spec file that would invoke Python’s zlib bindings similarly and hopefully demonstrate irregularities in compressed message sizes across architectures, but I didn’t reproduce anything interesting in a few tries, and I’m not prepared to invest a great deal of time in digging through aiohttp to try to reproduce this outside of its test suite.
The root cause isn’t resolved, but I’m preparing to skip the affected test on s390x in https://src.fedoraproject.org/rpms/python-aiohttp/pull-request/58, which will resolve the FTBFS.
FEDORA-2026-ada8dc0c53 (python-aiohttp-3.13.3-4.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-ada8dc0c53
FEDORA-2026-ada8dc0c53 (python-aiohttp-3.13.3-4.fc44) has been pushed to the Fedora 44 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2026-66cb8ecfc2 (python-aiohttp-3.13.3-4.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2026-66cb8ecfc2
FEDORA-2026-66cb8ecfc2 has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-66cb8ecfc2` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-66cb8ecfc2 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2026-66cb8ecfc2 (python-aiohttp-3.13.3-4.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.