Bug 1902297 - python-joblib fails to build with Python 3.10: AssertionError: assert 'ipython-input' in '__main__--builddir-build-BUILDROOT-python-joblib-0'
Summary: python-joblib fails to build with Python 3.10: AssertionError: assert 'ipytho...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-joblib
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Sergio Pascual
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.10
TreeView+ depends on / blocked
 
Reported: 2020-11-27 16:10 UTC by Tomáš Hrnčiar
Modified: 2020-12-18 00:55 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-18 00:55:11 UTC
Type: Bug


Attachments (Terms of Use)

Description Tomáš Hrnčiar 2020-11-27 16:10:32 UTC
python-joblib fails to build with Python 3.10.0a2.

=================================== FAILURES ===================================
_________ test_parallel_call_cached_function_defined_in_jupyter[True] __________

tmpdir = local('/tmp/pytest-of-mockbuild/pytest-0/test_parallel_call_cached_func0')
call_before_reducing = True

    @parametrize("call_before_reducing", [True, False])
    def test_parallel_call_cached_function_defined_in_jupyter(
        tmpdir, call_before_reducing
    ):
        # Calling an interactively defined memory.cache()'d function inside a
        # Parallel call used to clear the existing cache related to the said
        # function (https://github.com/joblib/joblib/issues/1035)
    
        # This tests checks that this is no longer the case.
    
        # TODO: test that the cache related to the function cache persists across
        # ipython sessions (provided that no code change were made to the
        # function's source)?
    
        # The first part of the test makes the necessary low-level calls to emulate
        # the definition of a function in an jupyter notebook cell. Joblib has
        # some custom code to treat functions defined specifically in jupyter
        # notebooks/ipython session -- we want to test this code, which requires
        # the emulation to be rigorous.
        for session_no in [0, 1]:
            ipython_cell_source = '''
            def f(x):
                return x
            '''
    
            ipython_cell_id = '<ipython-input-{}-000000000000>'.format(session_no)
    
            exec(
                compile(
                    textwrap.dedent(ipython_cell_source),
                    filename=ipython_cell_id,
                    mode='exec'
                )
            )
            # f is now accessible in the locals mapping - but for some unknown
            # reason, f = locals()['f'] throws a KeyError at runtime, we need to
            # bind locals()['f'] to a different name in the local namespace
            aliased_f = locals()['f']
            aliased_f.__module__ = "__main__"
    
            # Preliminary sanity checks, and tests checking that joblib properly
            # identified f as an interactive function defined in a jupyter notebook
            assert aliased_f(1) == 1
            assert aliased_f.__code__.co_filename == ipython_cell_id
    
            memory = Memory(location=tmpdir.strpath, verbose=0)
            cached_f = memory.cache(aliased_f)
    
            assert len(os.listdir(tmpdir / 'joblib')) == 1
            f_cache_relative_directory = os.listdir(tmpdir / 'joblib')[0]
>           assert 'ipython-input' in f_cache_relative_directory
E           AssertionError: assert 'ipython-input' in '__main__--builddir-build-BUILDROOT-python-joblib-0'

joblib/test/test_memory.py:178: AssertionError
_________ test_parallel_call_cached_function_defined_in_jupyter[False] _________

tmpdir = local('/tmp/pytest-of-mockbuild/pytest-0/test_parallel_call_cached_func1')
call_before_reducing = False

    @parametrize("call_before_reducing", [True, False])
    def test_parallel_call_cached_function_defined_in_jupyter(
        tmpdir, call_before_reducing
    ):
        # Calling an interactively defined memory.cache()'d function inside a
        # Parallel call used to clear the existing cache related to the said
        # function (https://github.com/joblib/joblib/issues/1035)
    
        # This tests checks that this is no longer the case.
    
        # TODO: test that the cache related to the function cache persists across
        # ipython sessions (provided that no code change were made to the
        # function's source)?
    
        # The first part of the test makes the necessary low-level calls to emulate
        # the definition of a function in an jupyter notebook cell. Joblib has
        # some custom code to treat functions defined specifically in jupyter
        # notebooks/ipython session -- we want to test this code, which requires
        # the emulation to be rigorous.
        for session_no in [0, 1]:
            ipython_cell_source = '''
            def f(x):
                return x
            '''
    
            ipython_cell_id = '<ipython-input-{}-000000000000>'.format(session_no)
    
            exec(
                compile(
                    textwrap.dedent(ipython_cell_source),
                    filename=ipython_cell_id,
                    mode='exec'
                )
            )
            # f is now accessible in the locals mapping - but for some unknown
            # reason, f = locals()['f'] throws a KeyError at runtime, we need to
            # bind locals()['f'] to a different name in the local namespace
            aliased_f = locals()['f']
            aliased_f.__module__ = "__main__"
    
            # Preliminary sanity checks, and tests checking that joblib properly
            # identified f as an interactive function defined in a jupyter notebook
            assert aliased_f(1) == 1
            assert aliased_f.__code__.co_filename == ipython_cell_id
    
            memory = Memory(location=tmpdir.strpath, verbose=0)
            cached_f = memory.cache(aliased_f)
    
            assert len(os.listdir(tmpdir / 'joblib')) == 1
            f_cache_relative_directory = os.listdir(tmpdir / 'joblib')[0]
>           assert 'ipython-input' in f_cache_relative_directory
E           AssertionError: assert 'ipython-input' in '__main__--builddir-build-BUILDROOT-python-joblib-0'

joblib/test/test_memory.py:178: AssertionError

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01797296-python-joblib/

For all our attempts to build python-joblib with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-joblib/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.


Note You need to log in before you can comment on or make changes to this bug.