Bug 2062338 - python-calcephpy fails to build with Python 3.11: error: invalid use of incomplete typedef PyFrameObject {aka struct _frame}
Summary: python-calcephpy fails to build with Python 3.11: error: invalid use of inco...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-calcephpy
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Mattia Verga
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON3.11
TreeView+ depends on / blocked
 
Reported: 2022-03-09 15:11 UTC by Tomáš Hrnčiar
Modified: 2022-03-24 18:04 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-23 10:56:34 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Tomáš Hrnčiar 2022-03-09 15:11:57 UTC
python-calcephpy fails to build with Python 3.11.0a6.

gcc -I../../src -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -Isrc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -Isrc -fPIC -I/builddir/build/BUILD/calcephpy-3.5.1/src -I/usr/include/python3.11 -c calcephpy.c -o build/temp.linux-x86_64-3.11/calcephpy.o
calcephpy.c: In function ‘__Pyx_PyBytes_Equals’:
calcephpy.c:29656:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
29656 |             hash1 = ((PyBytesObject*)s1)->ob_shash;
      |             ^~~~~
In file included from /usr/include/python3.11/bytesobject.h:62,
                 from /usr/include/python3.11/Python.h:50,
                 from calcephpy.c:40:
/usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
    7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
      |                                   ^~~~~~~~
calcephpy.c:29657:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
29657 |             hash2 = ((PyBytesObject*)s2)->ob_shash;
      |             ^~~~~
/usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
    7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
      |                                   ^~~~~~~~
calcephpy.c: In function ‘__Pyx_AddTraceback’:
calcephpy.c:615:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
  615 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
      |                                                              ^~
calcephpy.c:34011:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
34011 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/gcc' failed with exit code 1
make[2]: *** [Makefile:814: calcephpy_c_target] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/calcephpy-3.5.1/pythonapi/src'
make[1]: *** [Makefile:452: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/calcephpy-3.5.1/pythonapi'
make: *** [Makefile:501: all-recursive] Error 1
Traceback (most recent call last):
  File "/builddir/build/BUILD/calcephpy-3.5.1/setup.py", line 139, in <module>
    setup(
    ^^^^^^
  File "/usr/lib/python3.11/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/core.py", line 148, in setup
    dist.run_commands()
    ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
    ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
    ^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
    ^^^^^^^^^^^^^
  File "/builddir/build/BUILD/calcephpy-3.5.1/setup.py", line 84, in run
    self.run_command('build_clib')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
    ^^^^^^^^^^^^^
  File "/builddir/build/BUILD/calcephpy-3.5.1/setup.py", line 135, in run
    check_call(['make'], env=env)           
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2.

The PyFrameObject structure member has been moved to the internal C API headers.

While the documentation notes that the PyFrameObject fields are subject to change at any time, they have been stable for a long time and were used in several popular extensions.

In Python 3.11, the frame struct was reorganized to allow performance optimizations. Some fields were removed entirely, as they were details of the old implementation.

PyFrameObject fields:
    f_back: use PyFrame_GetBack().
    f_blockstack: removed.
    f_builtins: use PyObject_GetAttrString((PyObject*)frame, "f_builtins").
    f_code: use PyFrame_GetCode().
    f_gen: removed.
    f_globals: use PyObject_GetAttrString((PyObject*)frame, "f_globals").
    f_iblock: removed.
    f_lasti: use PyObject_GetAttrString((PyObject*)frame, "f_lasti"). Code using f_lasti with PyCode_Addr2Line() must use PyFrame_GetLineNumber() instead.
    f_lineno: use PyFrame_GetLineNumber()
    f_locals: use PyObject_GetAttrString((PyObject*)frame, "f_locals").
    f_stackdepth: removed.
    f_state: no public API (renamed to f_frame.f_state).
    f_trace: no public API.
    f_trace_lines: use PyObject_GetAttrString((PyObject*)frame, "f_trace_lines") (it also be modified).
    f_trace_opcodes: use PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes") (it also be modified).
    f_localsplus: no public API (renamed to f_frame.localsplus).
    f_valuestack: removed.

The Python frame object is now created lazily. A side effect is that the f_back member must not be accessed directly, since its value is now also computed lazily. The PyFrame_GetBack() function must be called instead.

https://docs.python.org/3.11/whatsnew/3.11.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.11/fedora-rawhide-x86_64/03638500-python-calcephpy/

For all our attempts to build python-calcephpy with Python 3.11, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/package/python-calcephpy/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.11:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/

Let us know here if you have any questions.

Python 3.11 is planned to be included in Fedora 37. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.11.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Mattia Verga 2022-03-09 16:56:56 UTC
I've reported to upstream. (by email, no bug tracker)

Comment 2 Mattia Verga 2022-03-17 17:52:59 UTC
I'm confused. From the instructions in the COPR repository:
"""
If you need to test a change in a Fedora package in this copr, open pull request at src.fedoraproject.org and obsereve the results in:

https://copr.fedorainfracloud.org/coprs/g/python/python3.11/builds/?dirname=python3.11:pr:<pr number here>
"""

To what repo I should open a pull request to run a test build?

Comment 4 Mattia Verga 2022-03-19 09:53:47 UTC
Build is still failing after applying a patch from upstream:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/builds/?dirname=python3.11:pr:1

I got this from upstream maintainer:

"""
You may still have an error with my new files.

I think that the last version of cython has a problem.

I reports this as an issue:
https://github.com/cython/cython/issues/4681

Another project has the same issue:
https://www.mail-archive.com/freebsd-pkg-fallout@freebsd.org/msg1906195.html


It should be related to

https://bugs.python.org/issue40421#msg414314
https://github.com/faster-cpython/ideas/issues/309
"""

As I understand, I need to wait for a newer release of Cython.

Comment 5 Tomáš Hrnčiar 2022-03-23 10:56:34 UTC
With patched Cython, calcephpy builds. Thank you for the help Mattia. I think, we can close this.

https://copr.fedorainfracloud.org/coprs/g/python/python3.11/build/3814446/

Comment 6 Mattia Verga 2022-03-23 17:23:56 UTC
(In reply to Tomáš Hrnčiar from comment #5)
> With patched Cython, calcephpy builds. Thank you for the help Mattia. I
> think, we can close this.
> 
> https://copr.fedorainfracloud.org/coprs/g/python/python3.11/build/3814446/

So, does the unpatched calcephpy build with the patched Cython? The calceph patch that upstream provided me isn't needed, it is?

Comment 7 Mattia Verga 2022-03-23 17:30:23 UTC
Uh, wait: the build that completed fine
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/build/3814446/
shows it was made with 
python3-Cython         x86_64  0.29.311-1.fc37

While previous builds 
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/builds/?dirname=python3.11:pr:1
were done with
python3-Cython         x86_64  3.0.0a10-1.fc36

Is that correct??

Comment 8 Tomáš Hrnčiar 2022-03-24 07:10:06 UTC
(In reply to Mattia Verga from comment #6)
> (In reply to Tomáš Hrnčiar from comment #5)
> > With patched Cython, calcephpy builds. Thank you for the help Mattia. I
> > think, we can close this.
> > 
> > https://copr.fedorainfracloud.org/coprs/g/python/python3.11/build/3814446/
> 
> So, does the unpatched calcephpy build with the patched Cython? The calceph
> patch that upstream provided me isn't needed, it is?

I am sorry, I thought that your patch was already merged.

I looked at old logs and it shows that calcephpy started to fail with 6th alpha of 3.11. With patched Cython (0.29.311 I used yesterday) it builds even without the patch so the problem probably wasn't in cacelphpy but in broken Cython. I am sorry for wasting your time, I shouldn't open this bugzilla.

Comment 9 Mattia Verga 2022-03-24 18:04:34 UTC
No problem at all, I was just confused about the relation between python 3.11 and Cython 3.11. I thought the upgrade to python 3.11 would have implied python3-Cython 3.0.0a10.


Note You need to log in before you can comment on or make changes to this bug.