Bug 2068421

Summary: python-executing fails to build with Python 3.11: InvocationError: self.get_executing(inspect.currentframe().f_back)
Product: [Fedora] Fedora Reporter: Tomáš Hrnčiar <thrnciar>
Component: python-executingAssignee: Roman Inflianskas <rominf>
Status: CLOSED RAWHIDE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: epel-packagers-sig, lbalhar, mhroncok, paul.wouters, rominf, thrnciar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-22 11:49:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2016048, 2069709    

Description Tomáš Hrnčiar 2022-03-25 09:15:24 UTC
python-executing fails to build with Python 3.11.0a6.

+ /usr/bin/python3 -m tox --current-env -q --recreate -e py311
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-executing-0.8.2-2.fc37.x86_64/usr/lib/python3.11/site-packages/executing/executing.py", line 317, in executing
    args = executing_cache[key]
           ~~~~~~~~~~~~~~~^^^^^
KeyError: (<code object <module> at 0x7fb5361790d0, file "/builddir/build/BUILD/executing-0.8.2/tests/test_main.py", line 1>, 140416273322192, 552)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/builddir/build/BUILD/executing-0.8.2/tests/test_main.py", line 682, in <module>
    assert tester([1, 2, 3]) == [1, 2, 3]
           ^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/executing-0.8.2/tests/utils.py", line 40, in __call__
    ex = self.get_executing(inspect.currentframe().f_back)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/executing-0.8.2/tests/utils.py", line 28, in get_executing
    return Source.executing(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILDROOT/python-executing-0.8.2-2.fc37.x86_64/usr/lib/python3.11/site-packages/executing/executing.py", line 372, in executing
    args = find(source=cls.for_frame(frame), retry_cache=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILDROOT/python-executing-0.8.2-2.fc37.x86_64/usr/lib/python3.11/site-packages/executing/executing.py", line 349, in find
    node_finder = NodeFinder(frame, stmts, tree, lasti)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILDROOT/python-executing-0.8.2-2.fc37.x86_64/usr/lib/python3.11/site-packages/executing/executing.py", line 626, in __init__
    raise RuntimeError(op_name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CALL
ERROR: InvocationError for command /builddir/build/BUILD/executing-0.8.2/.tox/py311/bin/python tests/test_main.py (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py311: commands failed
error: Bad exit status from /var/tmp/rpm-tmp.m2JoFZ (%check)
    Bad exit status from /var/tmp/rpm-tmp.m2JoFZ (%check)

Upstream has WIP PR for Python 3.11 suport.

The PyFrameObject structure member has been moved to the internal C API headers.

While the documentation notes that the PyFrameObject fields are subject to change at any time, they have been stable for a long time and were used in several popular extensions.

In Python 3.11, the frame struct was reorganized to allow performance optimizations. Some fields were removed entirely, as they were details of the old implementation.

PyFrameObject fields:
    f_back: use PyFrame_GetBack().
    f_blockstack: removed.
    f_builtins: use PyObject_GetAttrString((PyObject*)frame, "f_builtins").
    f_code: use PyFrame_GetCode().
    f_gen: removed.
    f_globals: use PyObject_GetAttrString((PyObject*)frame, "f_globals").
    f_iblock: removed.
    f_lasti: use PyObject_GetAttrString((PyObject*)frame, "f_lasti"). Code using f_lasti with PyCode_Addr2Line() must use PyFrame_GetLineNumber() instead.
    f_lineno: use PyFrame_GetLineNumber()
    f_locals: use PyObject_GetAttrString((PyObject*)frame, "f_locals").
    f_stackdepth: removed.
    f_state: no public API (renamed to f_frame.f_state).
    f_trace: no public API.
    f_trace_lines: use PyObject_GetAttrString((PyObject*)frame, "f_trace_lines") (it also be modified).
    f_trace_opcodes: use PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes") (it also be modified).
    f_localsplus: no public API (renamed to f_frame.localsplus).
    f_valuestack: removed.

The Python frame object is now created lazily. A side effect is that the f_back member must not be accessed directly, since its value is now also computed lazily. The PyFrame_GetBack() function must be called instead.

https://docs.python.org/3.11/whatsnew/3.11.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.11/fedora-rawhide-x86_64/03840142-python-executing/

For all our attempts to build python-executing with Python 3.11, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/package/python-executing/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.11:
https://copr.fedorainfracloud.org/coprs/g/python/python3.11/

Let us know here if you have any questions.

Python 3.11 is planned to be included in Fedora 37. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.11.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Roman Inflianskas 2022-03-25 15:02:17 UTC
Hello. Unfortunately I'm not qualified enough to handle this (I'm not super familiar with CPython internals) and I see that there is some discussion on GitHub and PR is not yet accepted. I would postpone this for now hoping that the author of the library address this.

Comment 2 Miro Hrončok 2022-03-25 18:08:38 UTC
> I see that there is some discussion on GitHub and PR is not yet accepted...

Could you share the links here, please?

Comment 3 Roman Inflianskas 2022-03-25 19:35:11 UTC
The link is attached to the issue, but I can copy-paste it here: https://github.com/alexmojaki/executing/pull/31

Comment 4 Lumír Balhar 2022-03-25 21:11:43 UTC
I've pinged there so it'll hopefully move forward soon. There is no rush. Tomáš can also try to disable the failing test and see whether the broken functionality is actually needed by any of the dependant packages.

Comment 5 Miro Hrončok 2022-03-25 23:53:38 UTC
(In reply to Roman Inflianskas from comment #3)
> The link is attached to the issue, but I can copy-paste it here:
> https://github.com/alexmojaki/executing/pull/31

Thanks, I've missed that.

Comment 6 Miro Hrončok 2022-06-22 11:49:14 UTC
Closing this in bulk as it built with Python 3.11. If this needs to remain open for a followup, feel free to reopen, I won't close in bulk again.