Bug 1792990 - python-fpylll fails to build with Python 3.9: TypeError: type() takes 1 or 3 arguments
Summary: python-fpylll fails to build with Python 3.9: TypeError: type() takes 1 or 3 ...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-fpylll
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Paulo Andrade
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1788506
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2020-01-20 13:19 UTC by Miro Hrončok
Modified: 2020-05-29 17:22 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-29 17:22:31 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Miro Hrončok 2020-01-20 13:19:16 UTC
python-fpylll fails to build with Python 3.9.0a2.

___________________________________ test_cvp ___________________________________
    def test_cvp():
        for m, n in dimensions:
            A = make_integer_matrix(m, n)
            A = LLL.reduction(A)
            M = GSO.Mat(A)
            M.update_gso()
            t = list(make_integer_matrix(n, n)[0])
            v0 = CVP.closest_vector(A, t)
    
            E = Enumeration(M)
            _, v1 = E.enumerate(0, A.nrows, 2, 40, M.from_canonical(t))[0]
>           v1 = IntegerMatrix.from_iterable(1, A.nrows, map(lambda x: int(round(x)), v1))
test_cvp.py:25: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   TypeError: type() takes 1 or 3 arguments
src/fpylll/fplll/integer_matrix.pyx:425: TypeError
_________________________________ test_gso_io __________________________________
    def test_gso_io():
        for int_type in int_types:
            for m, n in dimensions:
                if m <= 2 or n <= 2:
                    continue
    
                A = make_integer_matrix(m, n, int_type=int_type)
                v = list(A[0])
                LLL.reduction(A)
    
                for float_type in float_types:
                    M = GSO.Mat(copy(A), float_type=float_type)
                    M.update_gso()
                    w = M.babai(v)
>                   v_ = IntegerMatrix.from_iterable(1, m, w) * A
test_gso.py:109: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   TypeError: type() takes 1 or 3 arguments
src/fpylll/fplll/integer_matrix.pyx:425: TypeError
_________________________________ test_lll_lll _________________________________
    def test_lll_lll():
        for m, n in dimensions:
            A = make_integer_matrix(m, n)
            for int_type in int_types:
>               AA = IntegerMatrix.from_matrix(A, int_type=int_type)
test_lll.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   TypeError: type() takes 1 or 3 arguments
src/fpylll/fplll/integer_matrix.pyx:407: TypeError
________________________________ test_multisol _________________________________
    def test_multisol():
        A = make_integer_matrix()
        m = GSO.Mat(A)
        lll_obj = LLL.Reduction(m)
        lll_obj()
    
        solutions = []
        solutions = Enumeration(m, nr_solutions=200).enumerate(0, 27, 48.5, 0)
        assert len(solutions)== 126 / 2
        for _, sol in solutions:
>           sol = IntegerMatrix.from_iterable(1, A.nrows, map(lambda x: int(round(x)), sol))
test_multisol_enum.py:61: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   TypeError: type() takes 1 or 3 arguments
src/fpylll/fplll/integer_matrix.pyx:425: TypeError

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01161414-python-fpylll/

For all our attempts to build python-fpylll with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-fpylll/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Ben Cotton 2020-02-11 17:32:27 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

Comment 2 Jerry James 2020-02-25 18:50:32 UTC
When line 25 of test_cvp.py is reached:

        v1 = IntegerMatrix.from_iterable(1, A.nrows, map(lambda x: int(round(x)), v1))

control passes into Cython-generated code.  The source is src/fpylll/fplll/integer_matrix.pyx.  I set a breakpoint in gdb on the generated function, __pyx_pf_6fpylll_5fplll_14integer_matrix_13IntegerMatrix_4from_iterable.  The Cython definition looks like this (minus comments):

    @classmethod
    def from_iterable(cls, nrows, ncols, it, **kwds):
        A = cls(nrows, ncols, **kwds)
        A.set_iterable(it)
        return A

So cls is supposed to be a reference to the IntegerMatrix class, right?  It is, in python 3.8 and earlier, but not in python 3.9.  GDB says:

(gdb) print *__pyx_v_cls
$1 = {ob_base = {ob_base = {ob_refcnt = 54, 
      ob_type = 0x7ffff7dd52e0 <PyType_Type>}, ob_size = 0}, 
  tp_name = 0x7ffff7cdf35b "type", tp_basicsize = 872, tp_itemsize = 40, 
  tp_dealloc = 0x7ffff7c68030 <type_dealloc>, tp_vectorcall_offset = 400, 
  tp_getattr = 0x0, tp_setattr = 0x0, tp_as_async = 0x0, 
  tp_repr = 0x7ffff7c7cca0 <type_repr>, tp_as_number = 0x0, 
  tp_as_sequence = 0x0, tp_as_mapping = 0x0, 
  tp_hash = 0x7ffff7c03c30 <_Py_HashPointer>, 
  tp_call = 0x7ffff7bd79c0 <type_call>, tp_str = 0x7ffff7c16640 <object_str>, 
  tp_getattro = 0x7ffff7bd70e0 <type_getattro>, 
  tp_setattro = 0x7ffff7c547b0 <type_setattro>, tp_as_buffer = 0x0, 
  tp_flags = 2148293632, 
  tp_doc = 0x7ffff7d42600 "type(object_or_name, bases, dict)\ntype(object) -> the object's type\ntype(name, bases, dict) -> a new type", 
  tp_traverse = 0x7ffff7be5060 <type_traverse>, 
  tp_clear = 0x7ffff7c67f20 <type_clear>, 
  tp_richcompare = 0x7ffff7bf6e50 <object_richcompare>, 
  tp_weaklistoffset = 368, tp_iter = 0x0, tp_iternext = 0x0, 
  tp_methods = 0x7ffff7dbdee0, tp_members = 0x7ffff7dbdda0, 
  tp_getset = 0x7ffff7dbdc20, tp_base = 0x7ffff7dd5620 <PyBaseObject_Type>, 
  tp_dict = 0x7ffff78c2700, tp_descr_get = 0x0, tp_descr_set = 0x0, 
  tp_dictoffset = 264, tp_init = 0x7ffff7c45a60 <type_init>, 
  tp_alloc = 0x7ffff7bc31a0 <PyType_GenericAlloc>, 
  tp_new = 0x7ffff7bd7b00 <type_new>, 
  tp_free = 0x7ffff7bc5500 <PyObject_GC_Del>, 
  tp_is_gc = 0x7ffff7bc6370 <type_is_gc>, tp_bases = 0x7ffff78bf0a0, 
  tp_mro = 0x7ffff78c2e80, tp_cache = 0x0, tp_subclasses = 0x7ffff78299c0, 
  tp_weaklist = 0x7ffff78c0db0, tp_del = 0x0, tp_version_tag = 3, 
  tp_finalize = 0x0, tp_vectorcall = 0x0}

The code invokes that, expecting to get the IntegerMatrix constructor, and instead invokes the type() function.  The code generated by Cython in the python 3.8 and 3.9 cases differs only in strings that contain "3.8" in the first case and "3.9" in the second.  In Objects/classobject.c from python 3.9, there is a function named method_vectorcall starting at line 36.  We are going down the "then" branch of the conditional starting at line 48.  We set the first argument, which should be the class object, to "self", which is set on line 43:

    PyObject *self = PyMethod_GET_SELF(method);

GDB shows that self is the type() function.  The "method" object is:
$2 = {ob_base = {ob_refcnt = 1, ob_type = 0x7ffff7ddcb60 <PyMethod_Type>}, 
  im_func = <cython_function_or_method at remote 0x7ffff5c78f40>, 
  im_self = <type at remote 0x7ffff7dd52e0>, im_weakreflist = 0x0, 
  vectorcall = 0x7ffff7be5ed0 <method_vectorcall>}

where im_self is, indeed, the type() function.  The only place I can find where im_self is set is line 114 of Objects/classobject.c, in PyMethod_New:

PyObject *
PyMethod_New(PyObject *func, PyObject *self)
{
    if (self == NULL) {
        PyErr_BadInternalCall();
        return NULL;
    }
    PyMethodObject *im = PyObject_GC_New(PyMethodObject, &PyMethod_Type);
    if (im == NULL) {
        return NULL;
    }
    im->im_weakreflist = NULL;
    Py_INCREF(func);
    im->im_func = func;
    Py_INCREF(self);
    im->im_self = self;  // <-------- This is the only place I see where im_self is set.
    im->vectorcall = method_vectorcall;
    _PyObject_GC_TRACK(im);
    return (PyObject *)im;
}

I'm not sure how to follow the trail back from here.  I'm pretty certain that setting a breakpoint on PyMethod_New is going to drive me to madness.

Comment 3 Miro Hrončok 2020-02-25 19:20:00 UTC
This sounds like: https://bugzilla.redhat.com/show_bug.cgi?id=1793612#c4

Let me try to see whether this is fixed on Cython master.

Comment 4 Miro Hrončok 2020-02-25 19:36:07 UTC
It is.

class C:
    @classmethod
    def classmethod1(cls):
        pass

    @classmethod
    def classmethod2(cls):
        cls.classmethod1()


Cython 0.29.15:

$ python -c "import reproducer; reproducer.C.classmethod2()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "reproducer.pyx", line 8, in reproducer.C.classmethod2
    cls.classmethod1()
AttributeError: type object 'type' has no attribute 'classmethod1'


Cython master:

$ python -c "import reproducer; reproducer.C.classmethod2()"
(no output, no error)



OK, let me fins the commit that fixed this.

Comment 5 Miro Hrončok 2020-02-25 19:56:13 UTC
That is most likely https://github.com/cython/cython/pull/3106

Comment 6 Jerry James 2020-02-25 22:01:05 UTC
Thanks, Miro.  I'm glad you recognized that bug.

Comment 7 Miro Hrončok 2020-05-25 13:02:07 UTC
This comment is mass posted to all bugs blocking the Python 3.9 tracker, sorry if it is not 100 % relevant. When in doubt, please ask.


The Python 3.9 rebuild is in progress in a Koji side tag.

If you fix this bug, please don't rebuild the package in regular rawhide, but do it in the side tag with:

    $ fedpkg build --target=f33-python

The rebuild is progressing slowly and it is possible this package won't have all the required build dependencies yet. If that's the case, please just leave the fix committed and pushed and we will eventually rebuild it for you.

You are not asked to go and try rebuild all the missing dependencies yourself. If you know there is a bootstrap loop in the dependencies, let me know and we can untangle it together.

If you want to test your fix or reproduce the failure, you can still use the Copr repo mentioned in the initial comment of this bug: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Comment 8 Miro Hrončok 2020-05-29 07:11:58 UTC
Python 3.9 update: The f33-python side tag is currently being merged.

New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:

    $ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3

And build the packages normally after that.

Comment 9 Jerry James 2020-05-29 17:22:31 UTC
This was actually a cython bug and has since been fixed.  This package has been built in Rawhide post-merge.  Closing.


Note You need to log in before you can comment on or make changes to this bug.