awscli2 fails to build with Python 3.13.0a5. =================================== FAILURES =================================== ________________ TestEMRutils.test_which_with_existing_command _________________ [gw2] linux -- Python 3.13.0 /usr/bin/python3 self = <tests.unit.customizations.emr.test_emr_utils.TestEMRutils object at 0x7f2e4fae6180> def test_which_with_existing_command(self): > pythonPath = which('python') or which('python.exe') tests/unit/customizations/emr/test_emr_utils.py:20: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ program = 'python' def which(program): > for path in os.environ["PATH"].split(os.pathsep): E KeyError: 'PATH' ../../BUILDROOT/awscli2-2.15.31-1.fc41.x86_64/usr/lib/python3.13/site-packages/awscli/customizations/emr/emrutils.py:204: KeyError ______________ TestEMRutils.test_which_with_non_existing_command _______________ [gw2] linux -- Python 3.13.0 /usr/bin/python3 self = <tests.unit.customizations.emr.test_emr_utils.TestEMRutils object at 0x7f2e4fae4380> def test_which_with_non_existing_command(self): > path = which('klajsflklj') tests/unit/customizations/emr/test_emr_utils.py:24: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ program = 'klajsflklj' def which(program): > for path in os.environ["PATH"].split(os.pathsep): E KeyError: 'PATH' ../../BUILDROOT/awscli2-2.15.31-1.fc41.x86_64/usr/lib/python3.13/site-packages/awscli/customizations/emr/emrutils.py:204: KeyError =========================== short test summary info ============================ FAILED tests/unit/customizations/emr/test_emr_utils.py::TestEMRutils::test_which_with_existing_command FAILED tests/unit/customizations/emr/test_emr_utils.py::TestEMRutils::test_which_with_non_existing_command =========== 2 failed, 67596 passed, 7 skipped in 2720.40s (0:45:20) ============ https://docs.python.org/3.13/whatsnew/3.13.html For the build logs, see: https://copr-be.cloud.fedoraproject.org/results/@python/python3.13/fedora-rawhide-x86_64/07211089-awscli2/ For all our attempts to build awscli2 with Python 3.13, see: https://copr.fedorainfracloud.org/coprs/g/python/python3.13/package/awscli2/ Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.13: https://copr.fedorainfracloud.org/coprs/g/python/python3.13/ Let us know here if you have any questions. Python 3.13 is planned to be included in Fedora 41. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.13. A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon. We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.
*** Bug 2291489 has been marked as a duplicate of this bug. ***
Failed again in the main 3.13 rebuild, but with different test failures this time: =================================== FAILURES =================================== ______ TestDoesNotLeakMemory.test_create_single_paginator_memory_constant ______ [gw3] linux -- Python 3.13.0 /usr/bin/python3 self = <tests.functional.botocore.leak.test_resource_leaks.TestDoesNotLeakMemory testMethod=test_create_single_paginator_memory_constant> def test_create_single_paginator_memory_constant(self): self.cmd('create_paginator', 's3', 'list_objects') self.cmd('free_paginators') self.record_memory() for _ in range(100): self.cmd('create_paginator', 's3', 'list_objects') self.cmd('free_paginators') self.record_memory() start, end = self.memory_samples > self.assertTrue((end - start) < self.MAX_GROWTH_BYTES, (end - start)) E AssertionError: False is not true : 11010048 tests/functional/botocore/leak/test_resource_leaks.py:102: AssertionError _______ TestDoesNotLeakMemory.test_create_single_waiter_memory_constant ________ [gw3] linux -- Python 3.13.0 /usr/bin/python3 self = <tests.functional.botocore.leak.test_resource_leaks.TestDoesNotLeakMemory testMethod=test_create_single_waiter_memory_constant> def test_create_single_waiter_memory_constant(self): self.cmd('create_waiter', 's3', 'bucket_exists') self.cmd('free_waiters') self.record_memory() for _ in range(100): self.cmd('create_waiter', 's3', 'bucket_exists') self.cmd('free_waiters') self.record_memory() start, end = self.memory_samples > self.assertTrue((end - start) < self.MAX_GROWTH_BYTES, (end - start)) E AssertionError: False is not true : 10616832 tests/functional/botocore/leak/test_resource_leaks.py:76: AssertionError =========================== short test summary info ============================ FAILED tests/functional/botocore/leak/test_resource_leaks.py::TestDoesNotLeakMemory::test_create_single_paginator_memory_constant FAILED tests/functional/botocore/leak/test_resource_leaks.py::TestDoesNotLeakMemory::test_create_single_waiter_memory_constant =========== 2 failed, 67596 passed, 7 skipped in 1436.35s (0:23:56) ============ seems to be a memory leak check failing, not sure if it's really leaking memory or just the tests' assumptions need updating.
the test is from a bundled copy of botocore. python-botocore passed the 3.13 rebuild, but the memory leak test is actually disabled in the standalone package, for apparently-unrelated reasons: https://src.fedoraproject.org/rpms/python-botocore/c/b380603b4cb8aef479355b81004e053752ab85ec . I'm going to do a scratch build of python-botocore with the test re-enabled and see what happens.
hum, the test still fails in the same way described in the spec comment when run on standalone botocore: ==================================== ERRORS ==================================== _ ERROR at teardown of TestDoesNotLeakMemory.test_create_memory_clients_in_loop _ [gw6] linux -- Python 3.13.0 /usr/bin/python3 tests/__init__.py:197: in tearDown self.driver.stop() tests/__init__.py:239: in stop self.cmd('exit') tests/__init__.py:281: in cmd self.send_cmd(*cmd) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <tests.ClientDriver object at 0x7fff17ac78c0>, cmd = ('exit',) cmd_str = 'exit\n', cmd_bytes = b'exit\n' def send_cmd(self, *cmd): """Send a command and return immediately. This is a lower level method than cmd(). This method will instruct the cmd-runner process to execute a command, but this method will immediately return. You will need to use ``is_cmd_finished()`` to check that the command is finished. This method is useful if you want to record attributes about the process while an operation is occurring. For example, if you want to instruct the cmd-runner process to upload a 1GB file to S3 and you'd like to record the memory during the upload process, you can use send_cmd() instead of cmd(). """ cmd_str = ' '.join(cmd) + '\n' cmd_bytes = cmd_str.encode('utf-8') self._popen.stdin.write(cmd_bytes) > self._popen.stdin.flush() E BrokenPipeError: [Errno 32] Broken pipe tests/__init__.py:263: BrokenPipeError ----------------------------- Captured stderr call ----------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/python-botocore-1.34.125-build/botocore-1.34.125/tests/cmd-runner", line 72, in <module> import botocore.session ModuleNotFoundError: No module named 'botocore' so that doesn't help. will look into this all more tomorrow.
So I hacked up an environment where I can run the test_create_single_waiter_memory_constant test with an arbitrary number of runs in the loop and report back the memory use, and here's the results: 20 tries - 2883584 50 tries - 7340032 100 tries - 10616832 1000 tries - 11272192 10000 tries - 11272192 so I don't think it's truly leaking, it's just maxing out slightly higher than the test expects (the test cutoff is 10MiB or 10485760 bytes, we are coming in *just* above that with the default 100 tries, and maxing out just 6M higher). test_create_single_paginator_memory_constant is similar; with 10,000 tries it hits 11534336 bytes, just over the cutoff. I think we can just patch the cutoff a bit higher. I'm not sure exactly whether the increased usage is due to Python 3.13 or what, I might dig into that a bit in order to report this upstream.
Huh. So, I set up a mock env using the last Rawhide compose before Python 3.12 landed, and re-ran the same experiment. That should be pretty much the same package set *except* for Python 3.13 + rebuilds. The memory usage sure does max out a lot lower: 20 tries - 1048576 / 1048576 100 tries - 1703936 / 1703936 1000 tries - 1835008 / 1835008 10000 tries - 1835008 / 1835008 So we're maxing out around 1.83M, less than 20% of where we max out with Python 3.13. Not sure why the big difference, but I'll report it to upstream botocore in case they're interested. But for practical purposes, I guess I'll patch the max up to 20MiB so future builds pass tests.
Filed https://github.com/boto/botocore/issues/3205 , will patch the test headroom for now.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-cdfa8a41aa fixes this, waiting for signing.
Hello, Please note that this comment was generated automatically by https://pagure.io/releng/blob/main/f/scripts/ftbfs-fti/follow-policy.py If you feel that this output has mistakes, please open an issue at https://pagure.io/releng/ All subpackages of a package against which this bug was filled are now installable or removed from Fedora 41. Thanks for taking care of it!
*** Bug 2292242 has been marked as a duplicate of this bug. ***