Created attachment 1853539 [details]
build.log

file build.log too big, will only attach last 32768 bytes

Created attachment 1853540 [details]
root.log

file root.log too big, will only attach last 32768 bytes

Created attachment 1853541 [details]
state.log

Looking at https://koschei.fedoraproject.org/package/Cython -- this only fails on ppc64le.

Let's wait for gcc-12.0.1-0.3.fc36 before further analysis.

I've tested gcc-12.0.1-0.3.fc36 in mock on ppc64le-test.fedorainfracloud.org and it still fails :(

this will need rebuilt (or fixed) numpy I guess ...

Fixed how?

(In reply to Miro Hrončok from comment #7)
> Fixed how?

adapt to the standard IEEE "long double" if it's doing something special for ppc64le currently, but I believe it only needs a rebuild

Created attachment 1857080 [details]
ppc64le build.log with rebuilt numpy

When I rebuilt numpy with gcc-12.0.1-0.3.fc36 and use that numpy to build Cython with gcc-12.0.1-0.3.fc36, it still fails. build.log attached.

Created attachment 1857082 [details]
ppc64le build.log with rebuilt numpy

The actual build log is attached this time.

Next step: rebuild Cython without tests, rebuild numpy using that Cython, rebuild Cython with tests with that numpy.

No dice.

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

This poses a significant risk for timely Python 3.11 delivery in Fedora 37 - a month has passed and there is no clear way forward. Hence, rising severity.

Hi Miro,
any chance you could prepare a self-contained / reduced test case for the failure? I have close to zero clue what Cython is or how it works :-) It looks to me that only "memslice.schar_index_vars" test fails now, which should be something about signed char type in C. Unlikely related to the long double change, but IMO likely to related to the GCC rebase. Might be useful to try GCC 12 on F < 36 ...

and it might be a mis-compilation somewhere, after overriding -O2 with -O0 it passes all tests ... It makes it ever more important to have a self-contained test case.

this is with gcc-12.0.1-0.9.fc37.ppc64le

CCing Jakub as FYI

> any chance you could prepare a self-contained / reduced test case for the failure?


Without Cython? Very little chance. This is very very complex stuff.


> I have close to zero clue what Cython is or how it works :-)

Cython is a tool that compiles a Python-like language to C or C++ with Python C API. It is entirely possible that it makes assumptions about powerpc long double.


> Might be useful to try GCC 12 on F < 36 ...

How exactly would I do that? Unlike clang, the gcc maintainers don't appear to build alternate gcc versions for older Fedoras. Is there a copr repo I could use?


> after overriding -O2 with -O0 it passes all tests


In that case, it is possible that this is negatively impacted by https://fedoraproject.org/wiki/Changes/SetBuildFlagsBuildCheck -- I'll try a build that opts out.

A Fedora 37 build with %undefine _auto_set_build_flags:
https://koji.fedoraproject.org/koji/taskinfo?taskID=83504021


If that one succeeds, we still need to check if this has no negative impact on Python extension modules build with such Cython on ppc64le.



A Fedora 35 build with %set_build_falgs in %check:
https://koji.fedoraproject.org/koji/taskinfo?taskID=83504199

If this one fails the same way, we may assume it is the tests that don't work when CFLAGS is set with -O2 on ppc64le. That's worth fixing, but not critical.

(In reply to Miro Hrončok from comment #17)
> > any chance you could prepare a self-contained / reduced test case for the failure?
> 
> 
> Without Cython? Very little chance. This is very very complex stuff.

As a start I would like to reduce tests/memoryview/memslice.pyx to only contain the "schar" test plus a corresponding command line to run it.

> 
> 
> > I have close to zero clue what Cython is or how it works :-)
> 
> Cython is a tool that compiles a Python-like language to C or C++ with
> Python C API. It is entirely possible that it makes assumptions about
> powerpc long double.

Seems there is no problem with long double any more, but there was one initially IIRC. A "signed char" test fails, I believe it was failing in the mass rebuild run too.

Is there a way to save the produced intermediate C/C++ files somehow?

> 
> > Might be useful to try GCC 12 on F < 36 ...
> 
> How exactly would I do that? Unlike clang, the gcc maintainers don't appear
> to build alternate gcc versions for older Fedoras. Is there a copr repo I
> could use?

might be a matter of rebuilding f36+ gcc rpm on f34 or f35, but as decreasing the optimization level "fixes" it, then it looks as a regression in gcc 12
 
> 
> > after overriding -O2 with -O0 it passes all tests
> 
> 
> In that case, it is possible that this is negatively impacted by
> https://fedoraproject.org/wiki/Changes/SetBuildFlagsBuildCheck -- I'll try a
> build that opts out.

OK

(In reply to Miro Hrončok from comment #18)
> A Fedora 37 build with %undefine _auto_set_build_flags:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=83504021
> 
> 
> If that one succeeds, we still need to check if this has no negative impact
> on Python extension modules build with such Cython on ppc64le.


Nope, it failed.

> A Fedora 35 build with %set_build_falgs in %check:
> https://koji.fedoraproject.org/koji/taskinfo?taskID=83504199
> 
> If this one fails the same way, we may assume it is the tests that don't
> work when CFLAGS is set with -O2 on ppc64le. That's worth fixing, but not
> critical.

Nope, it suceeded.

When I try to build a more minimal reproducer, it fails for me with `IndexError: Out of bounds on buffer access (axis 0)`.



This is ppc64le with a fresh scratch build (tests disabled):


$ mock -r fedora-rawhide-ppc64le init
$ mock -r fedora-rawhide-ppc64le install gcc python3-devel https://kojipkgs.fedoraproject.org//work/tasks/5762/83505762/python3-Cython-0.29.26-2.fc37.ppc64le.rpm
$ mock -r fedora-rawhide-ppc64le --copyin reproducer.pyx /builddir
$ mock -r fedora-rawhide-ppc64le shell --unpriv 

<mock-chroot> sh-5.1$ python3
>>> import pyximport
>>> pyximport.install()
(None, <pyximport.pyximport.PyxImporter object at 0x7fffa5ce0cd0>)
>>> import reproducer
/usr/lib64/python3.10/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /builddir/reproducer.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
>>> C = reproducer.IntMockBuffer("C", range(300*300), (300, 300))
>>> reproducer.schar_index_vars(C, 1, 1, 5)
acquired C
reading
writing
validated
released C
301
>>> reproducer._read_int2d(C, 1, 1)
acquired C
released C
5
>>> reproducer.schar_index_vars(C, -1, 1, 6)
acquired C
reading
released C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "reproducer.pyx", line 197, in reproducer.schar_index_vars
    old_value = buf[i, j]
IndexError: Out of bounds on buffer access (axis 0)




This is my x86_64 mock with the latest regular build of Cython:

$ mock -r fedora-rawhide-x86_64 init
$ mock -r fedora-rawhide-x86_64 install gcc python3-devel Cython
$ mock -r fedora-rawhide-x86_64 --copyin reproducer.pyx /builddir
$ mock -r fedora-rawhide-x86_64 shell --unpriv 

<mock-chroot> sh-5.1$ python3
Python 3.10.2 (main, Feb  1 2022, 00:00:00) [GCC 12.0.1 20220129 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyximport
>>> pyximport.install()
(None, <pyximport.pyximport.PyxImporter object at 0x7fc05fdbe3e0>)
>>> import reproducer
/usr/lib64/python3.10/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /builddir/reproducer.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
>>> C = reproducer.IntMockBuffer("C", range(300*300), (300, 300))
>>> reproducer.schar_index_vars(C, 1, 1, 5)
acquired C
reading
writing
validated
released C
301
>>> reproducer._read_int2d(C, 1, 1)
acquired C
released C
5
>>> reproducer.schar_index_vars(C, -1, 1, 6)
acquired C
reading
writing
validated
released C
89701
>>> reproducer._read_int2d(C, -1, 1)
acquired C
released C
6
>>> reproducer.schar_index_vars(C, -1, -2, 7)
acquired C
reading
writing
validated
released C
89998

Created attachment 1863685 [details]
reproducer.pyx from my example

With the latest regular build of Cython on ppc64, I still get `IndexError: Out of bounds on buffer access (axis 0)` :(



To see the source, run `cythonize reproducer.pyx` and observe reproducer.c -- however it will be hard to read.

And this is ppc64le, Fedora 35, gcc 11:


<mock-chroot> sh-5.1$ python3
Python 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyximport
>>> pyximport.install()
(None, <pyximport.pyximport.PyxImporter object at 0x7fffba6023e0>)
>>> import reproducer
/usr/lib64/python3.10/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /builddir/reproducer.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
>>> C = reproducer.IntMockBuffer("C", range(300*300), (300, 300))
>>> reproducer.schar_index_vars(C, 1, 1, 5)
acquired C
reading
writing
validated
released C
301
>>> reproducer._read_int2d(C, 1, 1)
acquired C
released C
5
>>> reproducer.schar_index_vars(C, -1, 1, 6)
acquired C
reading
writing
validated
released C
89701
>>> reproducer._read_int2d(C, -1, 1)
acquired C
released C
6
>>> reproducer.schar_index_vars(C, -1, -2, 7)
acquired C
reading
writing
validated
released C
89998





tl;dr I cannot reproduce the same problem as in the test, but apparently, there is a different problem on ppc64le with GCC 12 that I can reproduce. It happens with the current build of Cython as well, when running with GCC 12 on runtime.

Could you attach the reproducer.pyx file?

I already did that.

Oh, so the attachment was marked as confidential, because, for some reason, Bugzilla assumed that's a reasonable default. I've made it public. Sorry about that.

Thanks :-)

Notes about debugging this Cython issue.

I suspect that the C code is generated from Cython put_buffer_lookup_code() compiler function at Cython/Compiler/Buffer.py:
---
def put_buffer_lookup_code(entry, index_signeds, index_cnames, directives,
                           pos, code, negative_indices, in_nogil_context):
    """
    Generates code to process indices and calculate an offset into
    a buffer. Returns a C string which gives a pointer which can be
    read from or written to at will (it is an expression so caller should
    store it in a temporary if it is used more than once).

    As the bounds checking can have any number of combinations of unsigned
    arguments, smart optimizations etc. we insert it directly in the function
    body. The lookup however is delegated to a inline function that is instantiated
    once per ndim (lookup with suboffsets tend to get quite complicated).

    entry is a BufferEntry
    """
    negative_indices = directives['wraparound'] and negative_indices

    if directives['boundscheck']:
        # Check bounds and fix negative indices.
        # We allocate a temporary which is initialized to -1, meaning OK (!).
        # If an error occurs, the temp is set to the index dimension the
        # error is occurring at.
        failed_dim_temp = code.funcstate.allocate_temp(PyrexTypes.c_int_type, manage_ref=False)
        code.putln("%s = -1;" % failed_dim_temp)
        for dim, (signed, cname, shape) in enumerate(zip(index_signeds, index_cnames, entry.get_buf_shapevars())):
            if signed != 0:
                # not unsigned, deal with negative index
                code.putln("if (%s < 0) {" % cname)
                if negative_indices:
                    code.putln("%s += %s;" % (cname, shape))
                    code.putln("if (%s) %s = %d;" % (
                        code.unlikely("%s < 0" % cname),
                        failed_dim_temp, dim))
                else:
                    code.putln("%s = %d;" % (failed_dim_temp, dim))
                code.put("} else ")
            # check bounds in positive direction
            if signed != 0:
                cast = ""
            else:
                cast = "(size_t)"
            code.putln("if (%s) %s = %d;" % (
                code.unlikely("%s >= %s%s" % (cname, cast, shape)),
                failed_dim_temp, dim))

        if in_nogil_context:
            code.globalstate.use_utility_code(raise_indexerror_nogil)
            func = '__Pyx_RaiseBufferIndexErrorNogil'
        else:
            code.globalstate.use_utility_code(raise_indexerror_code)
            func = '__Pyx_RaiseBufferIndexError'

        code.putln("if (%s) {" % code.unlikely("%s != -1" % failed_dim_temp))
        code.putln('%s(%s);' % (func, failed_dim_temp))
        code.putln(code.error_goto(pos))
        code.putln('}')
        code.funcstate.release_temp(failed_dim_temp)
    elif negative_indices:
        # Only fix negative indices.
        for signed, cname, shape in zip(index_signeds, index_cnames, entry.get_buf_shapevars()):
            if signed != 0:
                code.putln("if (%s < 0) %s += %s;" % (cname, cname, shape))

    return entry.generate_buffer_lookup_code(code, index_cnames)
---

which uses this Cython/Utility/Buffer.c code:
---
/////////////// BufferIndexError.proto ///////////////
static void __Pyx_RaiseBufferIndexError(int axis); /*proto*/

/////////////// BufferIndexError ///////////////
static void __Pyx_RaiseBufferIndexError(int axis) {
  PyErr_Format(PyExc_IndexError,
     "Out of bounds on buffer access (axis %d)", axis);
}
---

pyximport is an helper to build a Cython extension on "import something": build "something.pyx" into a Python extension module using Cython.

Command to build .pyx to .c: cython file.pyx

Use pyximport in debug mode, to see commands used to build a C extension:

    import distutils.debug
    distutils.debug.DEBUG = 1
    import distutils.log
    distutils.log.set_verbosity(2)

    import pyximport
    import pyximport.pyxbuild
    pyximport.pyxbuild.DEBUG = 1
    pyximport.install()

If I build the C extension using gcc -O0, I don't reproduce the bug. Using -O1 (or -O2), I reproduce the bug. It smells like a GCC 12 regression.

I wrote a short C reproducer which doesn't use Python nor Cython, no dependency at all.

I reported the GCC12 regression to GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=104839

Great job, Victor, thanks.

moving bug to gcc ...

(In reply to Miro Hrončok from comment #17)
> > Might be useful to try GCC 12 on F < 36 ...
> 
> How exactly would I do that? Unlike clang, the gcc maintainers don't appear
> to build alternate gcc versions for older Fedoras. Is there a copr repo I
> could use?

https://copr.fedorainfracloud.org/coprs/jwakely/gcc-latest/

(In reply to Jonathan Wakely from comment #32)
> (In reply to Miro Hrončok from comment #17)
> > > Might be useful to try GCC 12 on F < 36 ...
> > 
> > How exactly would I do that? Unlike clang, the gcc maintainers don't appear
> > to build alternate gcc versions for older Fedoras. Is there a copr repo I
> > could use?
> 
> https://copr.fedorainfracloud.org/coprs/jwakely/gcc-latest/

Oh, but I only enabled x86_64 :-(

and my local build with gcc-12.0.1-0.12.fc37.ppc64le confirms that Cython build succeeds now

FEDORA-2022-42ea499a7d has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d

FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-42ea499a7d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-d3234e63f1 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-d3234e63f1

FEDORA-2022-d3234e63f1 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-d3234e63f1`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-d3234e63f1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-42ea499a7d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

> Status: ON_QA → CLOSED

Great! I'm happy that this compiler bug is behind us ;-)

FEDORA-2022-d3234e63f1 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.