Bug 2271404 - awscli2 fails to build with Python 3.13: TestDoesNotLeakMemory fails
Summary: awscli2 fails to build with Python 3.13: TestDoesNotLeakMemory fails
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Fedora
Classification: Fedora
Component: awscli2
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Nikola Forró
QA Contact:
URL:
Whiteboard:
: 2291489 2292242 (view as bug list)
Depends On:
Blocks: F41FTBFS F41FailsToInstall PYTHON3.13 2291576
TreeView+ depends on / blocked
 
Reported: 2024-03-25 13:13 UTC by Karolina Surma
Modified: 2024-06-14 11:49 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2024-06-13 22:56:49 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Karolina Surma 2024-03-25 13:13:08 UTC
awscli2 fails to build with Python 3.13.0a5.
=================================== FAILURES ===================================
________________ TestEMRutils.test_which_with_existing_command _________________
[gw2] linux -- Python 3.13.0 /usr/bin/python3

self = <tests.unit.customizations.emr.test_emr_utils.TestEMRutils object at 0x7f2e4fae6180>

    def test_which_with_existing_command(self):
>       pythonPath = which('python') or which('python.exe')

tests/unit/customizations/emr/test_emr_utils.py:20: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

program = 'python'

    def which(program):
>       for path in os.environ["PATH"].split(os.pathsep):
E       KeyError: 'PATH'

../../BUILDROOT/awscli2-2.15.31-1.fc41.x86_64/usr/lib/python3.13/site-packages/awscli/customizations/emr/emrutils.py:204: KeyError
______________ TestEMRutils.test_which_with_non_existing_command _______________
[gw2] linux -- Python 3.13.0 /usr/bin/python3

self = <tests.unit.customizations.emr.test_emr_utils.TestEMRutils object at 0x7f2e4fae4380>

    def test_which_with_non_existing_command(self):
>       path = which('klajsflklj')

tests/unit/customizations/emr/test_emr_utils.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

program = 'klajsflklj'

    def which(program):
>       for path in os.environ["PATH"].split(os.pathsep):
E       KeyError: 'PATH'

../../BUILDROOT/awscli2-2.15.31-1.fc41.x86_64/usr/lib/python3.13/site-packages/awscli/customizations/emr/emrutils.py:204: KeyError
=========================== short test summary info ============================
FAILED tests/unit/customizations/emr/test_emr_utils.py::TestEMRutils::test_which_with_existing_command
FAILED tests/unit/customizations/emr/test_emr_utils.py::TestEMRutils::test_which_with_non_existing_command
=========== 2 failed, 67596 passed, 7 skipped in 2720.40s (0:45:20) ============

https://docs.python.org/3.13/whatsnew/3.13.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.13/fedora-rawhide-x86_64/07211089-awscli2/

For all our attempts to build awscli2 with Python 3.13, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.13/package/awscli2/

Testing and mass rebuild of packages is happening in copr.
You can follow these instructions to test locally in mock if your package builds with Python 3.13:
https://copr.fedorainfracloud.org/coprs/g/python/python3.13/

Let us know here if you have any questions.

Python 3.13 is planned to be included in Fedora 41.
To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.13.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires),
so if this package is required a lot, it's important for us to get it fixed soon.

We'd appreciate help from the people who know this package best,
but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Karolina Surma 2024-06-12 14:54:30 UTC
*** Bug 2291489 has been marked as a duplicate of this bug. ***

Comment 2 Adam Williamson 2024-06-13 06:00:19 UTC
Failed again in the main 3.13 rebuild, but with different test failures this time:

=================================== FAILURES ===================================
______ TestDoesNotLeakMemory.test_create_single_paginator_memory_constant ______
[gw3] linux -- Python 3.13.0 /usr/bin/python3
self = <tests.functional.botocore.leak.test_resource_leaks.TestDoesNotLeakMemory testMethod=test_create_single_paginator_memory_constant>
    def test_create_single_paginator_memory_constant(self):
        self.cmd('create_paginator', 's3', 'list_objects')
        self.cmd('free_paginators')
        self.record_memory()
        for _ in range(100):
            self.cmd('create_paginator', 's3', 'list_objects')
            self.cmd('free_paginators')
        self.record_memory()
        start, end = self.memory_samples
>       self.assertTrue((end - start) < self.MAX_GROWTH_BYTES, (end - start))
E       AssertionError: False is not true : 11010048
tests/functional/botocore/leak/test_resource_leaks.py:102: AssertionError
_______ TestDoesNotLeakMemory.test_create_single_waiter_memory_constant ________
[gw3] linux -- Python 3.13.0 /usr/bin/python3
self = <tests.functional.botocore.leak.test_resource_leaks.TestDoesNotLeakMemory testMethod=test_create_single_waiter_memory_constant>
    def test_create_single_waiter_memory_constant(self):
        self.cmd('create_waiter', 's3', 'bucket_exists')
        self.cmd('free_waiters')
        self.record_memory()
        for _ in range(100):
            self.cmd('create_waiter', 's3', 'bucket_exists')
            self.cmd('free_waiters')
        self.record_memory()
        start, end = self.memory_samples
>       self.assertTrue((end - start) < self.MAX_GROWTH_BYTES, (end - start))
E       AssertionError: False is not true : 10616832
tests/functional/botocore/leak/test_resource_leaks.py:76: AssertionError
=========================== short test summary info ============================
FAILED tests/functional/botocore/leak/test_resource_leaks.py::TestDoesNotLeakMemory::test_create_single_paginator_memory_constant
FAILED tests/functional/botocore/leak/test_resource_leaks.py::TestDoesNotLeakMemory::test_create_single_waiter_memory_constant
=========== 2 failed, 67596 passed, 7 skipped in 1436.35s (0:23:56) ============

seems to be a memory leak check failing, not sure if it's really leaking memory or just the tests' assumptions need updating.

Comment 3 Adam Williamson 2024-06-13 06:07:53 UTC
the test is from a bundled copy of botocore. python-botocore passed the 3.13 rebuild, but the memory leak test is actually disabled in the standalone package, for apparently-unrelated reasons: https://src.fedoraproject.org/rpms/python-botocore/c/b380603b4cb8aef479355b81004e053752ab85ec . I'm going to do a scratch build of python-botocore with the test re-enabled and see what happens.

Comment 4 Adam Williamson 2024-06-13 06:37:45 UTC
hum, the test still fails in the same way described in the spec comment when run on standalone botocore:

==================================== ERRORS ====================================
_ ERROR at teardown of TestDoesNotLeakMemory.test_create_memory_clients_in_loop _
[gw6] linux -- Python 3.13.0 /usr/bin/python3
tests/__init__.py:197: in tearDown
    self.driver.stop()
tests/__init__.py:239: in stop
    self.cmd('exit')
tests/__init__.py:281: in cmd
    self.send_cmd(*cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <tests.ClientDriver object at 0x7fff17ac78c0>, cmd = ('exit',)
cmd_str = 'exit\n', cmd_bytes = b'exit\n'
    def send_cmd(self, *cmd):
        """Send a command and return immediately.
    
        This is a lower level method than cmd().
        This method will instruct the cmd-runner process
        to execute a command, but this method will
        immediately return.  You will need to use
        ``is_cmd_finished()`` to check that the command
        is finished.
    
        This method is useful if you want to record attributes
        about the process while an operation is occurring.  For
        example, if you want to instruct the cmd-runner process
        to upload a 1GB file to S3 and you'd like to record
        the memory during the upload process, you can use
        send_cmd() instead of cmd().
    
        """
        cmd_str = ' '.join(cmd) + '\n'
        cmd_bytes = cmd_str.encode('utf-8')
        self._popen.stdin.write(cmd_bytes)
>       self._popen.stdin.flush()
E       BrokenPipeError: [Errno 32] Broken pipe
tests/__init__.py:263: BrokenPipeError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-botocore-1.34.125-build/botocore-1.34.125/tests/cmd-runner", line 72, in <module>
    import botocore.session
ModuleNotFoundError: No module named 'botocore'

so that doesn't help. will look into this all more tomorrow.

Comment 5 Adam Williamson 2024-06-13 15:30:23 UTC
So I hacked up an environment where I can run the test_create_single_waiter_memory_constant test with an arbitrary number of runs in the loop and report back the memory use, and here's the results:

   20 tries - 2883584
   50 tries - 7340032
  100 tries - 10616832
 1000 tries - 11272192
10000 tries - 11272192

so I don't think it's truly leaking, it's just maxing out slightly higher than the test expects (the test cutoff is 10MiB or 10485760 bytes, we are coming in *just* above that with the default 100 tries, and maxing out just 6M higher). test_create_single_paginator_memory_constant is similar; with 10,000 tries it hits 11534336 bytes, just over the cutoff. I think we can just patch the cutoff a bit higher. I'm not sure exactly whether the increased usage is due to Python 3.13 or what, I might dig into that a bit in order to report this upstream.

Comment 6 Adam Williamson 2024-06-13 15:53:43 UTC
Huh. So, I set up a mock env using the last Rawhide compose before Python 3.12 landed, and re-ran the same experiment. That should be pretty much the same package set *except* for Python 3.13 + rebuilds. The memory usage sure does max out a lot lower:

   20 tries - 1048576 / 1048576
  100 tries - 1703936 / 1703936
 1000 tries - 1835008 / 1835008
10000 tries - 1835008 / 1835008

So we're maxing out around 1.83M, less than 20% of where we max out with Python 3.13. Not sure why the big difference, but I'll report it to upstream botocore in case they're interested. But for practical purposes, I guess I'll patch the max up to 20MiB so future builds pass tests.

Comment 7 Adam Williamson 2024-06-13 17:10:35 UTC
Filed https://github.com/boto/botocore/issues/3205 , will patch the test headroom for now.

Comment 8 Adam Williamson 2024-06-13 22:38:13 UTC
https://bodhi.fedoraproject.org/updates/FEDORA-2024-cdfa8a41aa fixes this, waiting for signing.

Comment 9 Fedora Fails To Install 2024-06-13 22:56:49 UTC
Hello,

Please note that this comment was generated automatically by https://pagure.io/releng/blob/main/f/scripts/ftbfs-fti/follow-policy.py
If you feel that this output has mistakes, please open an issue at https://pagure.io/releng/

All subpackages of a package against which this bug was filled are now installable or removed from Fedora 41.

Thanks for taking care of it!

Comment 10 Karel Srot 2024-06-14 11:49:54 UTC
*** Bug 2292242 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.