Bug 1792062
| Summary: | python-jep fails to build with Python 3.9: daemon thread are not supported in subinterpreters | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Miro Hrončok <mhroncok> |
| Component: | python-jep | Assignee: | Raphael Groner <projects.rg> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rawhide | CC: | cstratak, igor.raits, mhroncok, projects.rg, pviktori, python-sig, vstinner |
| Target Milestone: | --- | Keywords: | FutureFeature, Patch, Reopened, Triaged, VerifiedUpstream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | https://src.fedoraproject.org/rpms/python-jep/pull-request/2 | ||
| Whiteboard: | |||
| Fixed In Version: | python-jep-3.9.1-1.fc33 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-01 18:52:58 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1885098 | ||
| Bug Blocks: | 1785415 | ||
Upstream issue: https://bugs.python.org/issue37266 Because of the reason given there, I doubt we can do anything as downstream to fix (CANTFIX). Upstream issue: https://bugs.python.org/issue37266 Because of the reason given there, I doubt we can *not* do anything as downstream to fix (CANTFIX). Raphael, I suggest you contact jep upstream about this. Victor, what can they do to fix this? > Victor, what can they do to fix this?
python-jep should use regular threads rather than daemon threads in subinterpreters.
Daemon threads in subinterpreters can crash at Python exit.
I reported the issue to jep: https://github.com/ninia/jep/issues/229 py4j as an alternative seems fixed, so we'd do as well for python-jep. This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle. Changing version to 32. Upstream has plans for a new major release 4.0, maybe without support for older python versions. https://github.com/ninia/jep/issues/202 Upstream patch available and promised for upcoming 3.9.1: https://github.com/ninia/jep/commit/a31d461c6cacc96de68d68320eaa83e19a45d0cc This comment is mass posted to all bugs blocking the Python 3.9 tracker, sorry if it is not 100 % relevant. When in doubt, please ask.
The Python 3.9 rebuild is in progress in a Koji side tag.
If you fix this bug, please don't rebuild the package in regular rawhide, but do it in the side tag with:
$ fedpkg build --target=f33-python
The rebuild is progressing slowly and it is possible this package won't have all the required build dependencies yet. If that's the case, please just leave the fix committed and pushed and we will eventually rebuild it for you.
You are not asked to go and try rebuild all the missing dependencies yourself. If you know there is a bootstrap loop in the dependencies, let me know and we can untangle it together.
If you want to test your fix or reproduce the failure, you can still use the Copr repo mentioned in the initial comment of this bug: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/
There is a new failure, see https://koji.fedoraproject.org/koji/taskinfo?taskID=45052014 test_restricted_classloader (test_import.TestImport) ... # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fff26424d40, pid=3660980, tid=0x00007fff06fdf100 # # JRE version: OpenJDK Runtime Environment (8.0_252-b01) (build 1.8.0_252-ea-b01) # Java VM: OpenJDK 64-Bit Server VM (25.252-b01 mixed mode linux-ppc64le compressed oops) # Problematic frame: # C [libpython3.9.so.1.0+0x74d40] PyEval_ReleaseLock+0x18 # # Core dump written. Default location: /builddir/build/BUILD/jep-3.9.0/core or core.3660980 # # An error report file with more information is saved as: # /builddir/build/BUILD/jep-3.9.0/hs_err_pid3660980.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # The jep crash may be related to https://github.com/ninia/jep/issues/229#issuecomment-633154702 Python 3.9 update: The f33-python side tag is currently being merged.
New builds in f33-python are no longer possible, but python3 is not yet updated to Python 3.9 in rawhide. You can check when Python is Python 3.9 with:
$ koji wait-repo f33-build --build python3.9-3.9.0~b1-3.fc3
And build the packages normally after that.
*** Bug 1842128 has been marked as a duplicate of this bug. *** There's now an issue with hard linkage to libpython without versioning. I tend to skip any support for python 3.9 for the moment. Might be of interest when the mass orphaning gets initiated due to FTI. Upstream is going to drop any support for backwards compatibility to python 2, introduced for announced (still in development) 4.0 major release. https://github.com/ninia/jep/commit/51c2ff3619247ae091eda45eb90e26376d3646cc Fixed in rawhide. Can you please retry? Added a patch from upstream to avoid daemon thread. But there's now an odd crash with (external) core dump from jdk, I'm adding all to build.log in case. Suspiction is to see some unhandled error/exceptin coming from inside libpython. https://koji.fedoraproject.org/koji/taskinfo?taskID=45486722 Please reopen if there's still an issue. Relevant snippet from build.log: Event: 0.309 Thread 0xb6246bf8 Exception <a 'java/security/PrivilegedActionException'> (0xa40fd240) thrown at [/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.252.b02-0.0.ea.fc33.arm/openjdk/hotspot/src/share/vm/prims/jvm.cpp, line 1499] Event: 1.024 Thread 0xb6246bf8 Exception <a 'jep/JepException'> (0xa40e0d68) thrown at [/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.252.b02-0.0.ea.fc33.arm/openjdk/hotspot/src/share/vm/prims/jni.cpp, line 710] Maybe that thrown JepException indicates a badly written unit test that fails to handle this invalid error. Reopening for further tracking. Upstream writes: "We will need to release Jep 3.9.1 as soon as Python 3.9 is released since Python 3.9 will not be compatible with any existing versions of Jep." New patch: https://github.com/ninia/jep/commit/83b7e654c3efb6b5fc906e877c6b0060eb8b83aa Hello, Please note that this comment was generated automatically. If you feel that this output has mistakes, please contact me via email (ignatenkobrain). All subpackages of a package against which this bug was filled are now installable or removed from Fedora 33. Thanks for taking care of it! (In reply to Raphael Groner from comment #19) > Reopening for further tracking. Upstream writes: > "We will need to release Jep 3.9.1 as soon as Python 3.9 is released since > Python 3.9 will not be compatible with any existing versions of Jep." > > New patch: > https://github.com/ninia/jep/commit/83b7e654c3efb6b5fc906e877c6b0060eb8b83aa Sent a PR with the patch. In koji rawhide it builds successfully. And of course forgot the link: https://src.fedoraproject.org/rpms/python-jep/pull-request/2 Thanks. Is your patch part of the new release 0.9.1 from upstream? See bug #1885098. In jep-3.9.1.tar.gz found at https://pypi.org/project/jep/#files, src/test/java/jep/test/TestSharedModulesThreads.java no longer contains "t.daemon = True". The downstream a31d461c6cacc96de68d68320eaa83e19a45d0cc.patch is no longer needed if the package is updated to jep 3.9.1. Fixed in rawhide. Please retry. Bodhi should add link, but slow performance currently. FEDORA-2021-35ae800ff3 has been pushed to the Fedora 33 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-35ae800ff3` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-35ae800ff3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2021-35ae800ff3 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report. |
python-jep fails to build with Python 3.9.0a2. FAIL: test_shared_modules_threads (test_shared_modules.TestSharedModules) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/jep-3.9.0/src/test/python/test_shared_modules.py", line 15, in test_shared_modules_threads jep_pipe(build_java_process_cmd('jep.test.TestSharedModulesThreads')) File "/usr/lib64/python3.9/contextlib.py", line 240, in helper return _GeneratorContextManager(func, args, kwds) File "/usr/lib64/python3.9/contextlib.py", line 83, in __init__ self.gen = func(*args, **kwds) File "/builddir/build/BUILD/jep-3.9.0/src/test/python/jep_pipe.py", line 36, in jep_pipe assert False, stderr AssertionError: b'Exception in thread "main" jep.JepException: <class \'RuntimeError\'>: daemon thread are not supported in subinterpreters\n\tat /usr/lib64/python3.9/threading.start(threading.py:858)\n\tat <string>.<module>(<string>:1)\n\tat jep.Jep.eval(Native Method)\n\tat jep.Jep.eval(Jep.java:451)\n\tat jep.test.TestSharedModulesThreads.main(TestSharedModulesThreads.java:53)\n' For the build logs, see: https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01149085-python-jep/ For all our attempts to build python-jep with Python 3.9, see: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-jep/ Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/ Let us know here if you have any questions. Python 3.9 will be included in Fedora 33. To make that update smoother, we're building Fedora packages with early pre-releases of Python 3.9. A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon. We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.