Bug 2271404

*** Bug 2291489 has been marked as a duplicate of this bug. ***

Failed again in the main 3.13 rebuild, but with different test failures this time:

=================================== FAILURES ===================================
______ TestDoesNotLeakMemory.test_create_single_paginator_memory_constant ______
[gw3] linux -- Python 3.13.0 /usr/bin/python3
self = <tests.functional.botocore.leak.test_resource_leaks.TestDoesNotLeakMemory testMethod=test_create_single_paginator_memory_constant>
    def test_create_single_paginator_memory_constant(self):
        self.cmd('create_paginator', 's3', 'list_objects')
        self.cmd('free_paginators')
        self.record_memory()
        for _ in range(100):
            self.cmd('create_paginator', 's3', 'list_objects')
            self.cmd('free_paginators')
        self.record_memory()
        start, end = self.memory_samples
>       self.assertTrue((end - start) < self.MAX_GROWTH_BYTES, (end - start))
E       AssertionError: False is not true : 11010048
tests/functional/botocore/leak/test_resource_leaks.py:102: AssertionError
_______ TestDoesNotLeakMemory.test_create_single_waiter_memory_constant ________
[gw3] linux -- Python 3.13.0 /usr/bin/python3
self = <tests.functional.botocore.leak.test_resource_leaks.TestDoesNotLeakMemory testMethod=test_create_single_waiter_memory_constant>
    def test_create_single_waiter_memory_constant(self):
        self.cmd('create_waiter', 's3', 'bucket_exists')
        self.cmd('free_waiters')
        self.record_memory()
        for _ in range(100):
            self.cmd('create_waiter', 's3', 'bucket_exists')
            self.cmd('free_waiters')
        self.record_memory()
        start, end = self.memory_samples
>       self.assertTrue((end - start) < self.MAX_GROWTH_BYTES, (end - start))
E       AssertionError: False is not true : 10616832
tests/functional/botocore/leak/test_resource_leaks.py:76: AssertionError
=========================== short test summary info ============================
FAILED tests/functional/botocore/leak/test_resource_leaks.py::TestDoesNotLeakMemory::test_create_single_paginator_memory_constant
FAILED tests/functional/botocore/leak/test_resource_leaks.py::TestDoesNotLeakMemory::test_create_single_waiter_memory_constant
=========== 2 failed, 67596 passed, 7 skipped in 1436.35s (0:23:56) ============

seems to be a memory leak check failing, not sure if it's really leaking memory or just the tests' assumptions need updating.

the test is from a bundled copy of botocore. python-botocore passed the 3.13 rebuild, but the memory leak test is actually disabled in the standalone package, for apparently-unrelated reasons: https://src.fedoraproject.org/rpms/python-botocore/c/b380603b4cb8aef479355b81004e053752ab85ec . I'm going to do a scratch build of python-botocore with the test re-enabled and see what happens.

hum, the test still fails in the same way described in the spec comment when run on standalone botocore:

==================================== ERRORS ====================================
_ ERROR at teardown of TestDoesNotLeakMemory.test_create_memory_clients_in_loop _
[gw6] linux -- Python 3.13.0 /usr/bin/python3
tests/__init__.py:197: in tearDown
    self.driver.stop()
tests/__init__.py:239: in stop
    self.cmd('exit')
tests/__init__.py:281: in cmd
    self.send_cmd(*cmd)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <tests.ClientDriver object at 0x7fff17ac78c0>, cmd = ('exit',)
cmd_str = 'exit\n', cmd_bytes = b'exit\n'
    def send_cmd(self, *cmd):
        """Send a command and return immediately.
    
        This is a lower level method than cmd().
        This method will instruct the cmd-runner process
        to execute a command, but this method will
        immediately return.  You will need to use
        ``is_cmd_finished()`` to check that the command
        is finished.
    
        This method is useful if you want to record attributes
        about the process while an operation is occurring.  For
        example, if you want to instruct the cmd-runner process
        to upload a 1GB file to S3 and you'd like to record
        the memory during the upload process, you can use
        send_cmd() instead of cmd().
    
        """
        cmd_str = ' '.join(cmd) + '\n'
        cmd_bytes = cmd_str.encode('utf-8')
        self._popen.stdin.write(cmd_bytes)
>       self._popen.stdin.flush()
E       BrokenPipeError: [Errno 32] Broken pipe
tests/__init__.py:263: BrokenPipeError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python-botocore-1.34.125-build/botocore-1.34.125/tests/cmd-runner", line 72, in <module>
    import botocore.session
ModuleNotFoundError: No module named 'botocore'

so that doesn't help. will look into this all more tomorrow.

So I hacked up an environment where I can run the test_create_single_waiter_memory_constant test with an arbitrary number of runs in the loop and report back the memory use, and here's the results:

   20 tries - 2883584
   50 tries - 7340032
  100 tries - 10616832
 1000 tries - 11272192
10000 tries - 11272192

so I don't think it's truly leaking, it's just maxing out slightly higher than the test expects (the test cutoff is 10MiB or 10485760 bytes, we are coming in *just* above that with the default 100 tries, and maxing out just 6M higher). test_create_single_paginator_memory_constant is similar; with 10,000 tries it hits 11534336 bytes, just over the cutoff. I think we can just patch the cutoff a bit higher. I'm not sure exactly whether the increased usage is due to Python 3.13 or what, I might dig into that a bit in order to report this upstream.

Huh. So, I set up a mock env using the last Rawhide compose before Python 3.12 landed, and re-ran the same experiment. That should be pretty much the same package set *except* for Python 3.13 + rebuilds. The memory usage sure does max out a lot lower:

   20 tries - 1048576 / 1048576
  100 tries - 1703936 / 1703936
 1000 tries - 1835008 / 1835008
10000 tries - 1835008 / 1835008

So we're maxing out around 1.83M, less than 20% of where we max out with Python 3.13. Not sure why the big difference, but I'll report it to upstream botocore in case they're interested. But for practical purposes, I guess I'll patch the max up to 20MiB so future builds pass tests.

Filed https://github.com/boto/botocore/issues/3205 , will patch the test headroom for now.

https://bodhi.fedoraproject.org/updates/FEDORA-2024-cdfa8a41aa fixes this, waiting for signing.

Hello,

Please note that this comment was generated automatically by https://pagure.io/releng/blob/main/f/scripts/ftbfs-fti/follow-policy.py
If you feel that this output has mistakes, please open an issue at https://pagure.io/releng/

All subpackages of a package against which this bug was filled are now installable or removed from Fedora 41.

Thanks for taking care of it!

*** Bug 2292242 has been marked as a duplicate of this bug. ***