Bug 1898116 - kdevelop-python fails to build with Python 3.10: error: node.h: No such file or directory [NEEDINFO]
Summary: kdevelop-python fails to build with Python 3.10: error: node.h: No such file ...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: kdevelop-python
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Orphan Owner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: F35FTBFS PYTHON3.10
TreeView+ depends on / blocked
 
Reported: 2020-11-16 12:48 UTC by Miro Hrončok
Modified: 2022-08-12 12:38 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
mhroncok: needinfo? (extras-orphan)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
KDE Software Compilation 438206 0 NOR UNCONFIRMED Fails to build against Python 3.10 2021-06-07 11:55:05 UTC

Description Miro Hrončok 2020-11-16 12:48:15 UTC
kdevelop-python fails to build with Python 3.10.0a2:

In file included from /builddir/build/BUILD/kdev-python-5.6.0/parser/astbuilder.cpp:31:
/builddir/build/BUILD/kdev-python-5.6.0/parser/python_header.h:33:10: fatal error: node.h: No such file or directory
   33 | #include "node.h"
      |          ^~~~~~~~
compilation terminated.


For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.10/fedora-rawhide-x86_64/01757099-kdevelop-python/

For all our attempts to build kdevelop-python with Python 3.10, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/kdevelop-python/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.10:
https://copr.fedorainfracloud.org/coprs/g/python/python3.10/

Let us know here if you have any questions.

Python 3.10 will be included in Fedora 35. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.10.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2020-11-16 15:02:14 UTC
This seem to be realated to the removal of the old parser: https://bugs.python.org/issue40939

Comment 2 Ben Cotton 2021-02-09 15:25:03 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 3 Miro Hrončok 2021-06-04 20:14:09 UTC
This is a mass-posted update. Sorry if it is not 100% accurate to this bugzilla.


The Python 3.10 rebuild is in progress in a Koji side tag. If you manage to fix the problem, please commit the fix in the rawhide branch, but don't build the package in regular rawhide.

You can either build the package in the side tag, with:

    $ fedpkg build --target=f35-python

Or you can the build and we will eventually build it for you.

Note that the rebuild is still in progress, so not all (build) dependencies of this package might be available right away.

Thanks.

See also https://fedoraproject.org/wiki/Changes/Python3.10

If you have general questions about the rebuild, please use this mailing list thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/G47SGOYIQLRDTWGOSLSWERZSSHXDEDH5/

Comment 4 Miro Hrončok 2021-06-07 22:58:49 UTC
The f35-python side tag has been merged to Rawhide. From now on, build as you would normally build.

Comment 5 Miro Hrončok 2021-07-26 09:08:47 UTC
I see there is no upstream response. How can we (the Python Maintainers) help you here?

Comment 6 Jan Grulich 2021-07-26 09:42:09 UTC
(In reply to Miro Hrončok from comment #5)
> I see there is no upstream response. How can we (the Python Maintainers)
> help you here?

Write a patch? I don't really know how much work it will be to support Python 3.10 in kdevelop-python, but I have zero knowlege in this area and won't be able to implement it myself. It's unfortunate we block you, but I don't know what to do about this besides waiting for upstream to fix it.

Comment 7 Miro Hrončok 2021-07-26 09:45:47 UTC
As much as we would like help as many package maintainers as possible, unfortunately we don't have the capacity to write hundreds of patches. I'll have a look at the code and see what it is trying to accomplish, possibly drafting a solution.

Comment 9 Miro Hrončok 2021-07-26 10:17:54 UTC
For reference. The following includes need to be removed from parser/python_header.h:

-#include "node.h"
-#include "Python-ast.h"
-#include "ast.h"
-#include "graminit.h"
-#include "grammar.h"
-#include "parsetok.h"


In parser/astbuilder.cpp PyParser_ASTFromString() needs to be replaced with Import_ImportModule("ast"), PyObject_CallMethod(..., "parse" ...).


I won't be able to finish this this week, but I'll talk to Victor Stinner and Petr Viktorin next week about documenting an upgrade path in https://docs.python.org/3.10/whatsnew/3.10.html#changes-in-the-c-api

Comment 10 Fedora Release Engineering 2021-08-01 04:22:24 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 35.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 36 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 34 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

Comment 11 Ben Cotton 2021-08-10 12:49:06 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 35 development cycle.
Changing version to 35.

Comment 12 Victor Stinner 2021-08-18 14:55:15 UTC
The parser API has been removed from the public C API. Many functions were renamed and "hidden" in the internal C API. For example, PyParser_ASTFromString() was renamed to _PyParser_ASTFromString() and moved to the pycore_parser.h internal C API. Same for PyArena_New() which was renamed to _PyArena_New().

kdev-python might get to retrieve most functions from the internal C API, but this is fragile. It was moved to the internal C API since this API changes often at each Python release, especially the exact AST tree. Each Python version adds new AST types, removes some AST types, and changes how expressions are stored in AST.

Another approach would be to use a completely different parser which would give a more reliable output.

Comment 13 Fedora Release Engineering 2021-08-22 04:22:17 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 35.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 36 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 34 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

Comment 14 Francis Herne 2021-09-07 08:53:24 UTC
Upstream kdev-python maintainer here.

I intend to add Python 3.10 support before the next kdev-python feature release, which should be in time for your purposes.

Every CPython release requires changes to kdev-python (Jan Grulich has linked to the last two cycles) so this is unsurprising.

We have always left supporting the next Python version until quite late in its release cycle, because the AST API is indeed prone to breaking changes even in late betas. We have a handy code-generation system that in most cases lets us adapt to AST changes quite easily.

This time there will be the added inconvenience of working around the API "hiding" that Victor Stinner mentioned. I wish I'd seen that issue earlier so I could request CPython upstream *not* do that and make our lives more difficult, but from previous experience they don't care at all about our use case anyway.

At some point it may be easier to maintain a downstream fork of the parser than work around the roadblocks that upstream likes to create, but honestly I don't have the motivation for that.

Comment 15 Fedora Release Engineering 2021-09-12 04:22:15 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 35.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 36 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 34 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

Comment 16 Victor Stinner 2021-09-24 10:19:04 UTC
Francis Herne: "This time there will be the added inconvenience of working around the API "hiding" that Victor Stinner mentioned. I wish I'd seen that issue earlier so I could request CPython upstream *not* do that and make our lives more difficult, but from previous experience they don't care at all about our use case anyway."

The C API still exists, but it has been moved to the internal C API which you are "not supposed to use". In practice, if you use it, it can break any time, there is even less backward compatibility warranties than on the private C API (names prefixed by "_Py"). The private C API has no backward compatibility warranty neither ;-)

The C API changes are related to a lot of compiler and AST changes introduced by the PEG parser:
https://www.python.org/dev/peps/pep-0617/

This issue is not a good place to raise awareness about your use case, you can open an issue at bugs.python.org or send an email to python-dev.

I don't know the ecosystem of parsers in Python projects. I don't know how black or pylint handle the new Python 3.10 grammar which now requires a parser more complicated than a LL(1) parser. By the way, the lib2to3 module is now deprecated in Python 3.11. I saw that black now uses a fork of this project and the fork supports Python 3.10 grammar, if I understood correctly.

Comment 17 Miro Hrončok 2021-09-24 13:04:29 UTC
This package has been orphaned, according to point 5 of https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/

If you wish to take over, you can pick it up at https://src.fedoraproject.org/rpms/kdevelop-python by clicking the "Take" button. If nobody picks it up, it will be retired and removed from the distribution in 6 weeks.

Comment 18 Miro Hrončok 2021-09-29 14:44:39 UTC
The final freeze for F35 starts in less than a week on Tuesday 2021-10-05. Setting severity to urgent.

Comment 19 Fedora Release Engineering 2021-10-03 04:22:10 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 35.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 36 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 34 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

Comment 20 Miro Hrončok 2021-10-04 12:52:11 UTC
> I intend to add Python 3.10 support before the next kdev-python feature release, which should be in time for your purposes.

Francis, is this still doable in time?

Comment 21 Fedora Release Engineering 2021-10-24 04:22:06 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 35.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 36 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 34 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

Comment 22 Fedora Release Engineering 2021-11-14 04:22:11 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 35.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 36 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 34 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-36/f-36-key-tasks.html

Comment 23 Fedora Release Engineering 2021-12-05 04:22:17 UTC
Dear Maintainer,

your package has an open Fails To Build From Source bug for Fedora 36.
Action is required from you.

If you can fix your package to build, perform a build in koji, and either create
an update in bodhi, or close this bug without creating an update, if updating is
not appropriate [1]. If you are working on a fix, set the status to ASSIGNED to
acknowledge this. If you have already fixed this issue, please close this Bugzilla report.

Following the policy for such packages [2], your package will be orphaned if
this bug remains in NEW state more than 8 weeks (not sooner than 2021-01-11).

A week before the mass branching of Fedora 37 according to the schedule [3],
any packages not successfully rebuilt at least on Fedora 35 will be
retired regardless of the status of this bug.

[1] https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/
[2] https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/
[3] https://fedorapeople.org/groups/schedule/f-37/f-37-key-tasks.html

Comment 24 Tomáš Trnka 2022-08-12 12:35:13 UTC
Actually, it's fairly easy to make kdevelop-python build against the python3.9 package instead of python3-devel. The end result works just fine with stock Fedora kdevelop (I've been using it extensively for several months now on multiple machines).

https://copr.fedorainfracloud.org/coprs/tootea/earlybird/package/kdevelop-python/
https://copr-dist-git.fedorainfracloud.org/cgit/tootea/earlybird/kdevelop-python.git/tree/?h=f36

Any chance we could get it back into Fedora this way? I guess we would only need a FESCo exception from the following rule:

> Fedora packages MUST NOT depend on other versions of the CPython interpreter than the current python3.
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_python_version_support

Is this feasible? Would a new review be necessary to un-orphan and revive the package?

Comment 25 Miro Hrončok 2022-08-12 12:38:14 UTC
An FPC exception would be needed.

And yes, a new review as well.


Note You need to log in before you can comment on or make changes to this bug.