Bug 1813075

Summary: pythran fails to build with ipython 7.12+
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: pythranAssignee: Miro Hrončok <mhroncok>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: cstratak, mhroncok, mplch, pviktori, python-sig, sguelton
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-18 13:42:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1750908, 1785415, 1803234    

Description Miro Hrončok 2020-03-12 21:37:31 UTC
pythran fails to build after ipython updated to 7.12

____________________ TestIpythonMagic.test_loadext_and_run _____________________
[gw1] linux -- Python 3.8.2 /usr/bin/python3
self = <pythran.tests.test_ipython.TestIpythonMagic testMethod=test_loadext_and_run>
    def test_loadext_and_run(self):
>       subprocess.check_call(['ipython', 'ipython_script.ipy'])
pythran/tests/test_ipython.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
popenargs = (['ipython', 'ipython_script.ipy'],), kwargs = {}, retcode = 2
cmd = ['ipython', 'ipython_script.ipy']
    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['ipython', 'ipython_script.ipy']' returned non-zero exit status 2.
/usr/lib64/python3.8/subprocess.py:364: CalledProcessError
----------------------------- Captured stderr call -----------------------------
[TerminalIPythonApp] WARNING | File 'ipython_script.ipy' doesn't exist
______________ TestIpythonMagic.test_loadext_and_run_timeit_twice ______________
[gw2] linux -- Python 3.8.2 /usr/bin/python3
self = <pythran.tests.test_ipython.TestIpythonMagic testMethod=test_loadext_and_run_timeit_twice>
    def test_loadext_and_run_timeit_twice(self):
>       subprocess.check_call(['ipython', 'ipython_script_timeit.ipy'])
pythran/tests/test_ipython.py:10: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
popenargs = (['ipython', 'ipython_script_timeit.ipy'],), kwargs = {}
retcode = 2, cmd = ['ipython', 'ipython_script_timeit.ipy']
    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['ipython', 'ipython_script_timeit.ipy']' returned non-zero exit status 2.
/usr/lib64/python3.8/subprocess.py:364: CalledProcessError
----------------------------- Captured stderr call -----------------------------
[TerminalIPythonApp] WARNING | File 'ipython_script_timeit.ipy' doesn't exist


See koschei:
https://koschei.fedoraproject.org/package/pythran?collection=f33
https://koschei.fedoraproject.org/package/pythran?collection=f32

The blocks the Python 3.9 rebuild.

For the build logs with Python 3.9, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01300678-pythran/

For all our attempts to build pythran with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/pythran/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Lumír Balhar 2020-03-13 08:41:51 UTC
I am not able to reproduce the issue.

Files downloaded from pythran's github:

# ll tests/
total 20
drwxr-xr-x. 2 root root 4096 Mar 13 08:36 __pycache__
-rw-r--r--. 1 root root  204 Mar 13 08:35 ipython_script.ipy
-rw-r--r--. 1 root root  466 Mar 13 08:35 ipython_script_timeit.ipy
-rw-r--r--. 1 root root  549 Mar 13 08:35 test_ipython.py

# pytest tests/
============================= test session starts ==============================
platform linux -- Python 3.8.2rc1, pytest-4.6.9, py-1.8.0, pluggy-0.13.0
rootdir: /tests
collected 2 items                                                              

tests/test_ipython.py FF                                                 [100%]

ModuleNotFoundError: No module named 'pythran'
ModuleNotFoundError: No module named 'pythran'

=========================== 2 failed in 0.98 seconds ===========================

The output is truncated but it's obvious that ipython loads both testing *.ipy files correctly and then fails because I don't have pythran installed.

Comment 2 Miro Hrončok 2020-03-13 08:55:34 UTC
> it's obvious that ipython loads both testing *.ipy files correctly

I am confused here. Obvious how? Sorry, but only thing that seems obvious (to me) from the above output is the "fails because I don't have pythran installed" part.

Comment 3 Lumír Balhar 2020-03-13 09:14:33 UTC
> I am confused here. Obvious how? Sorry, but only thing that seems obvious
> (to me) from the above output is the "fails because I don't have pythran
> installed" part.

In your report, the error message says that ipython cannot find/open the *.ipy files: "[TerminalIPythonApp] WARNING | File 'ipython_script_timeit.ipy' doesn't exist"

But in my try, ipython is able to open them, load them and then fail because the scripts try to load pythram extension which I don't have installed. So I am not able to reproduce the original issue with loading *.ipy scripts.

Is it more clear now?

BTW, the error might now be fixed upstream because the latest test_ipython.py uses os.path to determine the right path to *.ipy files but the version we have packaged does not: https://github.com/serge-sans-paille/pythran/blob/master/pythran/tests/test_ipython.py

Comment 4 Lumír Balhar 2020-03-13 09:16:23 UTC
Unfortunately, there is no new release after this fix: https://github.com/serge-sans-paille/pythran/commit/f5ae85881f47bfffe75ff2826e27d56514d69190

Comment 5 Miro Hrončok 2020-03-13 11:46:49 UTC
I'll try to backport that commit and see if it fixes the tests. Thanks.