Bug 1898157 - scipy fails to build with Python 3.10: TypeError: 'float' object cannot be interpreted as an integer
Summary: scipy fails to build with Python 3.10: TypeError: 'float' object cannot be in...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: scipy
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Nikola Forró
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.10
TreeView+ depends on / blocked
 
Reported: 2020-11-16 14:42 UTC by Miro Hrončok
Modified: 2020-11-27 08:17 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-27 08:17:05 UTC
Type: Bug


Attachments (Terms of Use)

Description Miro Hrončok 2020-11-16 14:42:24 UTC
scipy fails to build with Python 3.10.0a2.

[gw2] linux -- Python 3.10.0 /usr/bin/python3

self = <scipy.special.tests.test_basic.TestFactorialFunctions object at 0x7f49b2e2b490>

    def test_mixed_nan_inputs(self):
        x = np.array([np.nan, 1, 2, 3, np.nan])
        with suppress_warnings() as sup:
            sup.filter(DeprecationWarning, "Using factorial\\(\\) with floats is deprecated")
>           result = special.factorial(x, exact=True)

scipy/special/tests/test_basic.py:1829: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

n = array([nan,  1.,  2.,  3., nan]), exact = True

    def factorial(n, exact=False):
        """
        The factorial of a number or array of numbers.
    
        The factorial of non-negative integer `n` is the product of all
        positive integers less than or equal to `n`::
    
            n! = n * (n - 1) * (n - 2) * ... * 1
    
        Parameters
        ----------
        n : int or array_like of ints
            Input values.  If ``n < 0``, the return value is 0.
        exact : bool, optional
            If True, calculate the answer exactly using long integer arithmetic.
            If False, result is approximated in floating point rapidly using the
            `gamma` function.
            Default is False.
    
        Returns
        -------
        nf : float or int or ndarray
            Factorial of `n`, as integer or float depending on `exact`.
    
        Notes
        -----
        For arrays with ``exact=True``, the factorial is computed only once, for
        the largest input, with each other result computed in the process.
        The output dtype is increased to ``int64`` or ``object`` if necessary.
    
        With ``exact=False`` the factorial is approximated using the gamma
        function:
    
        .. math:: n! = \\Gamma(n+1)
    
        Examples
        --------
        >>> from scipy.special import factorial
        >>> arr = np.array([3, 4, 5])
        >>> factorial(arr, exact=False)
        array([   6.,   24.,  120.])
        >>> factorial(arr, exact=True)
        array([  6,  24, 120])
        >>> factorial(5, exact=True)
        120
    
        """
        if exact:
            if np.ndim(n) == 0:
                if np.isnan(n):
                    return n
                return 0 if n < 0 else math.factorial(n)
            else:
                n = asarray(n)
                un = np.unique(n).astype(object)
    
                # Convert to object array of long ints if np.int_ can't handle size
                if np.isnan(n).any():
                    dt = float
                elif un[-1] > 20:
                    dt = object
                elif un[-1] > 12:
                    dt = np.int64
                else:
                    dt = np.int_
    
                out = np.empty_like(n, dtype=dt)
    
                # Handle invalid/trivial values
                # Ignore runtime warning when less operator used w/np.nan
                with np.errstate(all='ignore'):
                    un = un[un > 1]
                    out[n < 2] = 1
                    out[n < 0] = 0
    
                # Calculate products of each range of numbers
                if un.size:
>                   val = math.factorial(un[0])
E                   TypeError: 'float' object cannot be interpreted as an integer

scipy/special/_basic.py:2352: TypeError




For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01756937-scipy/

For all our attempts to build scipy with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/scipy/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Victor Stinner 2020-11-24 14:43:56 UTC
I reported the issue to scipy: https://github.com/scipy/scipy/issues/13122

I proposed a fix upstream: https://github.com/scipy/scipy/pull/13121

Comment 2 Nikola Forró 2020-11-24 14:50:49 UTC
(In reply to Victor Stinner from comment #1)
> I reported the issue to scipy: https://github.com/scipy/scipy/issues/13122
> 
> I proposed a fix upstream: https://github.com/scipy/scipy/pull/13121

Thanks, but that only solves a subset of the issues, AFAICT. There are also multiple crashes during test runs, similar to this:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fbd50d9337a in vgetargskeywordsfast_impl () from /lib64/libpython3.10.so.1.0
#1  0x00007fbd50e1b60e in _PyArg_ParseStackAndKeywords_SizeT () from /lib64/libpython3.10.so.1.0
#2  0x00007fbd0c0f9f0d in zlib_Decompress_decompress () from /usr/lib64/python3.10/lib-dynload/zlib.cpython-310-x86_64-linux-gnu.so
#3  0x00007fbd0af746f4 in __Pyx_PyObject_CallOneArg ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#4  0x00007fbd0af77b5e in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream__fill_buffer.part.0 ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#5  0x00007fbd0af77de6 in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream_read_into ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#6  0x00007fbd0af916aa in __pyx_f_5scipy_2io_6matlab_10mio5_utils_10VarReader5_cread_full_tag.lto_priv.0 ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so
#7  0x00007fbd0af923ec in __pyx_pw_5scipy_2io_6matlab_10mio5_utils_10VarReader5_9read_full_tag.lto_priv.0 ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-1.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so
#8  0x00007fbd50d3f0f2 in method_vectorcall_NOARGS () from /lib64/libpython3.10.so.1.0
#9  0x00007fbd50d2e559 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#10 0x00007fbd50d3bb83 in function_code_fastcall () from /lib64/libpython3.10.so.1.0
#11 0x00007fbd50d2e559 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#12 0x00007fbd50d2cf2c in _PyEval_EvalCode () from /lib64/libpython3.10.so.1.0
#13 0x00007fbd50d3b8d6 in _PyFunction_Vectorcall () from /lib64/libpython3.10.so.1.0
#14 0x00007fbd50d2e559 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#15 0x00007fbd50d2cf2c in _PyEval_EvalCode () from /lib64/libpython3.10.so.1.0
#16 0x00007fbd50d3b8d6 in _PyFunction_Vectorcall () from /lib64/libpython3.10.so.1.0
#17 0x00007fbd50d2e2fb in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#18 0x00007fbd50d2cf2c in _PyEval_EvalCode () from /lib64/libpython3.10.so.1.0
#19 0x00007fbd50dabba5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.10.so.1.0
#20 0x00007fbd50dabb3d in PyEval_EvalCodeEx () from /lib64/libpython3.10.so.1.0
#21 0x00007fbd50dabaef in PyEval_EvalCode () from /lib64/libpython3.10.so.1.0
#22 0x00007fbd50dd340d in run_eval_code_obj () from /lib64/libpython3.10.so.1.0
#23 0x00007fbd50dd2456 in run_mod () from /lib64/libpython3.10.so.1.0
#24 0x00007fbd50d1b707 in PyRun_InteractiveOneObjectEx () from /lib64/libpython3.10.so.1.0
#25 0x00007fbd50d1bffb in PyRun_InteractiveLoopFlags () from /lib64/libpython3.10.so.1.0
#26 0x00007fbd50cb520c in PyRun_AnyFileExFlags.cold () from /lib64/libpython3.10.so.1.0
#27 0x00007fbd50cb3d42 in Py_RunMain.cold () from /lib64/libpython3.10.so.1.0
#28 0x00007fbd50d9ccfd in Py_BytesMain () from /lib64/libpython3.10.so.1.0
#29 0x00007fbd50a8eba2 in __libc_start_main () from /lib64/libc.so.6
#30 0x000055e51ae4909e in _start ()

Comment 3 Victor Stinner 2020-11-24 15:10:48 UTC
> Thanks, but that only solves a subset of the issues, AFAICT. There are also multiple crashes during test runs, (...)

Oh, I didn't know. Good to know.

Are you interested to investigate, or do you want me to have a look?

Comment 4 Victor Stinner 2020-11-25 13:48:03 UTC
My first PR was wrong. I wrote a second PR which has been merged into master:
https://github.com/scipy/scipy/commit/277dbab1612ae32f02038a6a0df7e061c927a5cc

It simply skips the test on Python 3.10. scipy is going to deprecate and then remove float support in scipy.special.factorial().

Comment 5 Nikola Forró 2020-11-25 16:50:56 UTC
(In reply to Victor Stinner from comment #3)
> Are you interested to investigate, or do you want me to have a look?

Sure, you can have a look, thanks.

All the crashes seem to have the same cause, I tracked it down to this so far:
https://github.com/scipy/scipy/blob/v1.5.4/scipy/io/matlab/streams.pyx#L137

Here is a simplified reproducer:
python3.10 -c 'import scipy.io.matlab.mio; scipy.io.matlab.mio.loadmat("scipy/io/matlab/tests/data/little_endian.mat")'

Comment 6 Victor Stinner 2020-11-25 17:01:35 UTC
> python3.10 -c 'import scipy.io.matlab.mio; scipy.io.matlab.mio.loadmat("scipy/io/matlab/tests/data/little_endian.mat")'

Thanks for the reproducer.

gdb traceback:

(gdb) where
#0  0x00007ffff7de337a in vgetargskeywordsfast_impl (args=0x1, nargs=0, kwargs=0x0, kwnames=<unknown at remote 0x7ffff583cea4>, parser=0x7ffff5843660 <_parser.19>, p_va=0x7fffffffd220, flags=2)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/getargs.c:2034
#1  0x00007ffff7e6b60e in _PyArg_ParseStackAndKeywords_SizeT (args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>, parser=parser@entry=0x7ffff5843660 <_parser.19>)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/getargs.c:1519
#2  0x00007ffff583cf0d in zlib_Decompress_decompress (self=0x7ffff45e1df0, cls=0x7fffffffd3e8, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Modules/clinic/zlibmodule.c.h:394
#3  0x00007ffff46276f4 in __Pyx_PyObject_CallOneArg ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#4  0x00007ffff462ab5e in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream__fill_buffer.part.0 ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#5  0x00007ffff462ade6 in __pyx_f_5scipy_2io_6matlab_7streams_15ZlibInputStream_read_into ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/streams.cpython-310-x86_64-linux-gnu.so
#6  0x00007ffff46446aa in __pyx_f_5scipy_2io_6matlab_10mio5_utils_10VarReader5_cread_full_tag.lto_priv.0 ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so
#7  0x00007ffff46453ec in __pyx_pw_5scipy_2io_6matlab_10mio5_utils_10VarReader5_9read_full_tag.lto_priv.0 ()
   from /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5_utils.cpython-310-x86_64-linux-gnu.so
#8  0x00007ffff7d8f0f2 in method_vectorcall_NOARGS (func=<method_descriptor at remote 0x7ffff467f630>, args=0x7ffff77513e0, nargsf=<optimized out>, kwnames=0x0)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Objects/descrobject.c:434
#9  0x00007ffff7d7e559 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff77513e0, callable=<method_descriptor at remote 0x7ffff467f630>, tstate=0x555555560cd0)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Include/cpython/abstract.h:114
#10 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7ffff77513e0, callable=<method_descriptor at remote 0x7ffff467f630>)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Include/cpython/abstract.h:123
#11 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555555560cd0) at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/ceval.c:5341
#12 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/ceval.c:3753
#13 0x00007ffff7d8bb83 in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7ffff7751240, for file /builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5.py, line 267, in read_var_header (self=<MatFile5Reader(mat_stream=<_io.BufferedReader at remo
te 0x7ffff77ab250>, dtypes={}, byte_order='<', struct_as_record=True, squeeze_me=False, chars_as_strings=True, mat_dtype=False, verify_compressed_data_integrity=True, simplify_cells=False, uint16_codec='utf-8', _file_reader=<scipy.io.matl
ab.mio5_utils.VarReader5 at remote 0x7ffff4f63a00>, _matrix_reader=<scipy.io.matlab.mio5_utils.VarReader5 at remote--Type <RET> for more, q to quit, c to continue without paging--
 0x7ffff4c15040>) at remote 0x7ffff781f730>, mdtype=15, byte_count=48, next_pos=184, stream=<scipy.io.matlab.streams.ZlibInputStream at remote 0x7ffff45daf20>, check_stream_limit=True), 
    tstate=0x555555560cd0) at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Include/internal/pycore_ceval.h:40
(...)

(gdb) py-bt
Traceback (most recent call first):
  File "/builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5.py", line 267, in read_var_header
    mdtype, byte_count = self._matrix_reader.read_full_tag()
  File "/builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio5.py", line 313, in get_variables
    hdr, next_position = self.read_var_header()
  File "/builddir/build/BUILDROOT/scipy-1.5.4-2.fc34.x86_64/usr/lib64/python3.10/site-packages/scipy/io/matlab/mio.py", line 224, in loadmat
    matfile_dict = MR.get_variables(variable_names)
  File "<string>", line 1, in <module>

(gdb) frame 0
#0  0x00007ffff7de337a in vgetargskeywordsfast_impl (args=0x1, nargs=0, kwargs=0x0, kwnames=<unknown at remote 0x7ffff583cea4>, parser=0x7ffff5843660 <_parser.19>, p_va=0x7fffffffd220, flags=2)
    at /usr/src/debug/python3.10-3.10.0~a2-1.fc34.x86_64/Python/getargs.c:2034
2034	    if (kwnames != NULL && !PyTuple_Check(kwnames)) {

(gdb) p *kwnames
$3 = {ob_refcnt = -8554210076332781581, ob_type = 0x41ff3145c2894cd2}


The zlib_Decompress_decompress() function is called with the wrong calling convention. It gets a kwnames object which is not a tuple of strings but looks like an invalid PyObject*.

In Python 3.10, this function is declared as:

#define ZLIB_DECOMPRESS_DECOMPRESS_METHODDEF    \
    {"decompress", (PyCFunction)(void(*)(void))zlib_Decompress_decompress, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, zlib_Decompress_decompress__doc__},

static PyObject *
zlib_Decompress_decompress(compobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
{
    PyObject *return_value = NULL;
    static const char * const _keywords[] = {"", "max_length", NULL};
    static _PyArg_Parser _parser = {"y*|n:decompress", _keywords, 0};
    Py_buffer data = {NULL, NULL};
    Py_ssize_t max_length = 0;

    if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser,
        &data, &max_length)) {
        goto exit;
    }
    return_value = zlib_Decompress_decompress_impl(self, cls, &data, max_length);

exit:
    /* Cleanup for data */
    if (data.obj) {
       PyBuffer_Release(&data);
    }

    return return_value;
}


My bet is an issue in Cython optimized functions which doesn't support properly METH_METHOD. The surprising part is that METH_METHOD was introduced in Python 3.9, not in Python 3.10.

The difference in Python 3.9, the function didn't use METH_METHOD:

#define ZLIB_DECOMPRESS_DECOMPRESS_METHODDEF    \
    {"decompress", (PyCFunction)(void(*)(void))zlib_Decompress_decompress, METH_FASTCALL|METH_KEYWORDS, zlib_Decompress_decompress__doc__},

static PyObject *
zlib_Decompress_decompress(compobject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
{
    ...
}

Comment 7 Victor Stinner 2020-11-25 17:04:03 UTC
About Python 3.10, the following calling convention:

    {"decompress", (PyCFunction)(void(*)(void))zlib_Decompress_decompress, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, zlib_Decompress_decompress__doc__},

comes from this Argument Clinic code:

/*[clinic input]
zlib.Decompress.decompress

    cls: defining_class
    data: Py_buffer
        The binary data to decompress.
    /
    max_length: Py_ssize_t = 0
        The maximum allowable length of the decompressed data.
        Unconsumed input data will be stored in
        the unconsumed_tail attribute.

...
[clinic start generated code]*/


The zlib module started to use "defining_class" argument type.

Comment 8 Victor Stinner 2020-11-25 17:09:44 UTC
> My bet is an issue in Cython optimized functions which doesn't support properly METH_METHOD.

I reported the issue to Cython:
https://github.com/cython/cython/issues/3917

scipy only works on Python 3.9 because zlib didn't use METH_METHOD flag. It started to use it in Python 3.10.

Comment 9 Nikola Forró 2020-11-25 18:41:21 UTC
Thanks Victor. I added your patch to skip the factorial test in Rawhide: https://src.fedoraproject.org/rpms/scipy/c/138e3931a60cebfa5c0d3163f9da90e8bf705ced?branch=master

Comment 10 Miro Hrončok 2020-11-27 08:01:53 UTC
With this: https://src.fedoraproject.org/rpms/Cython/pull-request/24

scipy builds!

Thank you all.


Note You need to log in before you can comment on or make changes to this bug.