Bug 1902297

Summary: python-joblib fails to build with Python 3.10: AssertionError: assert 'ipython-input' in '__main__--builddir-build-BUILDROOT-python-joblib-0'
Product: [Fedora] Fedora Reporter: Tomáš Hrnčiar <thrnciar>
Component: python-joblibAssignee: Sergio Pascual <sergio.pasra>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: besser82, igor.raits, mhroncok, sergio.pasra, thrnciar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-18 00:55:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1890881    

Description Tomáš Hrnčiar 2020-11-27 16:10:32 UTC
python-joblib fails to build with Python 3.10.0a2.

=================================== FAILURES ===================================
_________ test_parallel_call_cached_function_defined_in_jupyter[True] __________

tmpdir = local('/tmp/pytest-of-mockbuild/pytest-0/test_parallel_call_cached_func0')
call_before_reducing = True

    @parametrize("call_before_reducing", [True, False])
    def test_parallel_call_cached_function_defined_in_jupyter(
        tmpdir, call_before_reducing
    ):
        # Calling an interactively defined memory.cache()'d function inside a
        # Parallel call used to clear the existing cache related to the said
        # function (https://github.com/joblib/joblib/issues/1035)
    
        # This tests checks that this is no longer the case.
    
        # TODO: test that the cache related to the function cache persists across
        # ipython sessions (provided that no code change were made to the
        # function's source)?
    
        # The first part of the test makes the necessary low-level calls to emulate
        # the definition of a function in an jupyter notebook cell. Joblib has
        # some custom code to treat functions defined specifically in jupyter
        # notebooks/ipython session -- we want to test this code, which requires
        # the emulation to be rigorous.
        for session_no in [0, 1]:
            ipython_cell_source = '''
            def f(x):
                return x
            '''
    
            ipython_cell_id = '<ipython-input-{}-000000000000>'.format(session_no)
    
            exec(
                compile(
                    textwrap.dedent(ipython_cell_source),
                    filename=ipython_cell_id,
                    mode='exec'
                )
            )
            # f is now accessible in the locals mapping - but for some unknown
            # reason, f = locals()['f'] throws a KeyError at runtime, we need to
            # bind locals()['f'] to a different name in the local namespace
            aliased_f = locals()['f']
            aliased_f.__module__ = "__main__"
    
            # Preliminary sanity checks, and tests checking that joblib properly
            # identified f as an interactive function defined in a jupyter notebook
            assert aliased_f(1) == 1
            assert aliased_f.__code__.co_filename == ipython_cell_id
    
            memory = Memory(location=tmpdir.strpath, verbose=0)
            cached_f = memory.cache(aliased_f)
    
            assert len(os.listdir(tmpdir / 'joblib')) == 1
            f_cache_relative_directory = os.listdir(tmpdir / 'joblib')[0]
>           assert 'ipython-input' in f_cache_relative_directory
E           AssertionError: assert 'ipython-input' in '__main__--builddir-build-BUILDROOT-python-joblib-0'

joblib/test/test_memory.py:178: AssertionError
_________ test_parallel_call_cached_function_defined_in_jupyter[False] _________

tmpdir = local('/tmp/pytest-of-mockbuild/pytest-0/test_parallel_call_cached_func1')
call_before_reducing = False

    @parametrize("call_before_reducing", [True, False])
    def test_parallel_call_cached_function_defined_in_jupyter(
        tmpdir, call_before_reducing
    ):
        # Calling an interactively defined memory.cache()'d function inside a
        # Parallel call used to clear the existing cache related to the said
        # function (https://github.com/joblib/joblib/issues/1035)
    
        # This tests checks that this is no longer the case.
    
        # TODO: test that the cache related to the function cache persists across
        # ipython sessions (provided that no code change were made to the
        # function's source)?
    
        # The first part of the test makes the necessary low-level calls to emulate
        # the definition of a function in an jupyter notebook cell. Joblib has
        # some custom code to treat functions defined specifically in jupyter
        # notebooks/ipython session -- we want to test this code, which requires
        # the emulation to be rigorous.
        for session_no in [0, 1]:
            ipython_cell_source = '''
            def f(x):
                return x
            '''
    
            ipython_cell_id = '<ipython-input-{}-000000000000>'.format(session_no)
    
            exec(
                compile(
                    textwrap.dedent(ipython_cell_source),
                    filename=ipython_cell_id,
                    mode='exec'
                )
            )
            # f is now accessible in the locals mapping - but for some unknown
            # reason, f = locals()['f'] throws a KeyError at runtime, we need to
            # bind locals()['f'] to a different name in the local namespace
            aliased_f = locals()['f']
            aliased_f.__module__ = "__main__"
    
            # Preliminary sanity checks, and tests checking that joblib properly
            # identified f as an interactive function defined in a jupyter notebook
            assert aliased_f(1) == 1
            assert aliased_f.__code__.co_filename == ipython_cell_id
    
            memory = Memory(location=tmpdir.strpath, verbose=0)
            cached_f = memory.cache(aliased_f)
    
            assert len(os.listdir(tmpdir / 'joblib')) == 1
            f_cache_relative_directory = os.listdir(tmpdir / 'joblib')[0]
>           assert 'ipython-input' in f_cache_relative_directory
E           AssertionError: assert 'ipython-input' in '__main__--builddir-build-BUILDROOT-python-joblib-0'

joblib/test/test_memory.py:178: AssertionError

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01797296-python-joblib/

For all our attempts to build python-joblib with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-joblib/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.