Bug 1817770

Summary: xonsh fails to build with Python 3.9: unsupported operand type(s) for +: 'NoneType' and 'int'
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: xonshAssignee: Miro Hrončok <mhroncok>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 33CC: carmen, cstratak, igor.raits, jdulaney, kparal, lbalhar, lewk, mhroncok, mplch, python-sig
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedFreezeException
Fixed In Version: xonsh-0.9.21-1.fc34 xonsh-0.9.21-1.fc33 xonsh-0.9.21-1.eln108 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-14 23:44:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1822355    
Bug Blocks: 1766776, 1785415, 1803234, 1803235, 1868279    

Description Miro Hrončok 2020-03-26 23:35:45 UTC
xonsh fails to build with Python 3.9.0a5.

____________________ ERROR collecting tests/test_xonsh.xsh _____________________
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/pytest_plugin.py:40: in collect
    mod = importlib.import_module(self.fspath.purebasename)
/usr/lib64/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:786: in exec_module
    ???
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23413: in get_code
    code = execer.compile(src, glbs=ctx, locs=ctx)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23101: in compile
    tree = self.parse(input, ctx, mode=mode, filename=filename, transform=transform)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23061: in parse
    tree, input = self._parse_ctx_free(input, mode=mode, filename=filename)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23182: in _parse_ctx_free
    tree = self.parser.parse(
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/parsers/base.py:543: in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/ply/ply/yacc.py:335: in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/ply/ply/yacc.py:1122: in parseopt_notrack
    p.callable(pslice)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/parsers/base.py:2346: in p_atom_expr
    p[0] = self.apply_trailers(p[1], p[2])
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/parsers/base.py:2340: in apply_trailers
    assert False
E   assert False
_________________ ERROR collecting tests/test_lib/test_os.xsh __________________
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/pytest_plugin.py:40: in collect
    mod = importlib.import_module(self.fspath.purebasename)
/usr/lib64/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
    ???
<frozen importlib._bootstrap>:1007: in _find_and_load
    ???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:786: in exec_module
    ???
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23413: in get_code
    code = execer.compile(src, glbs=ctx, locs=ctx)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23101: in compile
    tree = self.parse(input, ctx, mode=mode, filename=filename, transform=transform)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:23075: in parse
    tree = self.ctxtransformer.ctxvisit(
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:8313: in ctxvisit
    node = self.visit(node)
/usr/lib64/python3.9/ast.py:395: in visit
    return visitor(node)
/usr/lib64/python3.9/ast.py:471: in generic_visit
    value = self.visit(value)
/usr/lib64/python3.9/ast.py:395: in visit
    return visitor(node)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:8528: in visit_FunctionDef
    self.generic_visit(node)
/usr/lib64/python3.9/ast.py:471: in generic_visit
    value = self.visit(value)
/usr/lib64/python3.9/ast.py:395: in visit
    return visitor(node)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:8506: in visit_With
    self.generic_visit(node)
/usr/lib64/python3.9/ast.py:471: in generic_visit
    value = self.visit(value)
/usr/lib64/python3.9/ast.py:395: in visit
    return visitor(node)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:8506: in visit_With
    self.generic_visit(node)
/usr/lib64/python3.9/ast.py:471: in generic_visit
    value = self.visit(value)
/usr/lib64/python3.9/ast.py:395: in visit
    return visitor(node)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:8428: in visit_Expr
    newnode = self.try_subproc_toks(node)
../../BUILDROOT/xonsh-0.9.13-2.fc33.x86_64/usr/lib/python3.9/site-packages/xonsh/__amalgam__.py:8380: in try_subproc_toks
    increment_lineno(newnode, n=node.lineno - 1)
/usr/lib64/python3.9/ast.py:230: in increment_lineno
    child.end_lineno = getattr(child, 'end_lineno', 0) + n
E   TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!


For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01321133-xonsh/

For all our attempts to build xonsh with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/xonsh/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2020-05-25 13:02:34 UTC
This comment is mass posted to all bugs blocking the Python 3.9 tracker, sorry if it is not 100 % relevant. When in doubt, please ask.


The Python 3.9 rebuild is in progress in a Koji side tag.

If you fix this bug, please don't rebuild the package in regular rawhide, but do it in the side tag with:

    $ fedpkg build --target=f33-python

The rebuild is progressing slowly and it is possible this package won't have all the required build dependencies yet. If that's the case, please just leave the fix committed and pushed and we will eventually rebuild it for you.

You are not asked to go and try rebuild all the missing dependencies yourself. If you know there is a bootstrap loop in the dependencies, let me know and we can untangle it together.

If you want to test your fix or reproduce the failure, you can still use the Copr repo mentioned in the initial comment of this bug: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Comment 2 Miro Hrončok 2020-05-29 07:12:16 UTC
Python 3.9 update: The f33-python side tag is currently being merged.

New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:

    $ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3

And build the packages normally after that.

Comment 3 Miro Hrončok 2020-05-31 10:44:58 UTC
*** Bug 1842163 has been marked as a duplicate of this bug. ***

Comment 5 Lumír Balhar 2020-06-02 10:42:01 UTC
Upstream is slowly working on this (https://github.com/xonsh/xonsh/pull/3588) but there will be also changes in CPython related to ast issues (https://github.com/python/cpython/pull/20312/files).

Comment 6 Igor Raits 2020-06-21 18:06:59 UTC
Hello,

This is the second reminder (step 4 from https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/#_package_removal_for_long_standing_ftbfs_and_fti_bugs).

If you know about this problem and are planning on fixing it, please acknowledge so by setting the bug status to ASSIGNED. If you don't have time to maintain this package, consider orphaning it, so maintainers of dependent packages realize the problem.

Comment 7 Carmen Bianca Bakker 2020-06-25 06:22:06 UTC
Apologies, assigning myself. Was in the midst of finishing my studies. Things have calmed down now.

Comment 8 Ben Cotton 2020-08-11 13:15:26 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Comment 9 Miro Hrončok 2020-08-22 10:30:56 UTC
Hello. This comment is mass posted to all 16 bugzillas for packages that failed to rebuild with Python 3.9 (and, in most cases, also fail to install on Fedora 33 and 34). Apologies if some of the suggestions or questions don't apply 100%.

Fedora 33 Beta Freeze starts next week on Tuesday, 2020-08-25. I haven't spammed this bugzilla because it is in ASSIGNED, or I've seen upstream has been notified.

Please consider what to do here. Is this failure critical? For example, can a test be temporarily skipped to make this package build? Is it possible to fix or workaround the failure before Fedora 33 Beta? Or before Fedora 33 Final? If not, would it make sense to retire the package (even temporarily) before the release to unblock updates from Fedora 31 or Fedora 32?

In case you need any assistance with a Python-related problem with the build, don't hesitate to let me know and I'll try to have a look.

Thank you!

Comment 10 Miro Hrončok 2020-08-24 10:06:28 UTC
*** Bug 1871642 has been marked as a duplicate of this bug. ***

Comment 11 Charalampos Stratakis 2020-09-07 17:57:07 UTC
There is some work done by upstream to make the package functional with 3.9: https://github.com/xonsh/xonsh/pull/3736/files

However the PR to fully address the issues is still in draft state: https://github.com/xonsh/xonsh/pull/3588

Comment 12 Miro Hrončok 2020-09-10 17:02:37 UTC
https://github.com/xonsh/xonsh/pull/3736/files is included in the 0.9.21 release. Marking this dependent on the update bug.

https://github.com/xonsh/xonsh/pull/3588 is a draft but older than the above. I'll see the actual errors when updating to 0.9.21.

Comment 14 Fedora Blocker Bugs Application 2020-09-10 18:46:49 UTC
Proposed as a Freeze Exception for 33-beta by Fedora user churchyard using the blocker tracking app because:

 The package does not install without the fix.

Comment 15 Fedora Update System 2020-09-10 19:15:08 UTC
FEDORA-2020-05da50e552 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 16 Fedora Update System 2020-09-10 19:20:31 UTC
FEDORA-2020-afca245174 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-afca245174

Comment 17 Fedora Update System 2020-09-11 20:18:03 UTC
FEDORA-2020-afca245174 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-afca245174`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-afca245174

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Kamil Páral 2020-09-14 14:45:11 UTC
Accepted as a Beta freeze exception:
https://pagure.io/fedora-qa/blocker-review/issue/81

Comment 19 Fedora Update System 2020-09-14 23:44:22 UTC
FEDORA-2020-afca245174 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Fedora Update System 2021-01-13 02:26:11 UTC
FEDORA-2021-0bf2083471 has been pushed to the Fedora ELN stable repository.
If problem still persists, please make note of it in this bug report.