Bug 1706238 - Python 3.8 forbids any forks with java process as additional interpreter / python-jep FTBFS with Python 3.8
Summary: Python 3.8 forbids any forks with java process as additional interpreter / py...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-jep
Version: rawhide
Hardware: Unspecified
OS: Unspecified
low
high
Target Milestone: ---
Assignee: Raphael Groner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1745894
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-05-03 21:04 UTC by Charalampos Stratakis
Modified: 2019-08-31 15:34 UTC (History)
2 users (show)

Fixed In Version: python-jep-3.9.0-1.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-31 15:34:58 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Full log from Copr (221.96 KB, text/plain)
2019-05-03 21:04 UTC, Charalampos Stratakis
no flags Details

Description Charalampos Stratakis 2019-05-03 21:04:11 UTC
Created attachment 1562747 [details]
Full log from Copr

python-jep-3.8.2-2 fails to build with Python 3.8 due to various test failures, all of which produce:

RuntimeError: fork not supported for subinterpreters

Snippet:

ERROR: test_exception_cause (test_exceptions.TestExceptions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/jep-3.8.2/src/test/python/test_exceptions.py", line 84, in test_exception_cause
    jep_pipe(build_java_process_cmd('jep.test.TestExceptionCause'))
  File "/usr/lib64/python3.8/contextlib.py", line 240, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/usr/lib64/python3.8/contextlib.py", line 83, in __init__
    self.gen = func(*args, **kwds)
  File "/builddir/build/BUILD/jep-3.8.2/src/test/python/jep_pipe.py", line 32, in jep_pipe
    p = subprocess.Popen(argv, stdout=PIPE, stderr=PIPE)
  File "/usr/lib64/python3.8/subprocess.py", line 830, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.8/subprocess.py", line 1583, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
RuntimeError: fork not supported for subinterpreters


Full log attached

Comment 2 Raphael Groner 2019-05-11 12:02:58 UTC
Fixed in rawhide. Please retest.
https://src.fedoraproject.org/rpms/python-jep/c/b04f360bf3702db89466c8e18d8a34ddff7d74f2?branch=master

Comment 3 Raphael Groner 2019-05-11 12:16:00 UTC
No, there's need for another patch. Having to investigate even deeper.
Maybe the fork is forbidden because we run python as main interpreter inside mock? Meaning we can't use fork at all?

Comment 4 Miro Hrončok 2019-05-11 12:56:38 UTC
Of course you can use fork in mock. You cannot use fork in a subinterpreter.
I suggest you bring this upstream.

Comment 5 Raphael Groner 2019-05-11 13:30:31 UTC
Upstream is aware about the issue with subprocess. Personally, I don't feel competent enough with the internals of Cython to have a good base to discuss with upstream. Let's hope upstream can fix it when python 3.8 is in a stable state and adds official support.

> @unittest.skipIf(sys.platform.startswith("win"), "subprocess complications on Windows")

$ grep -l 'subprocess complications' src/test/python/*.py
src/test/python/test_exceptions.py
src/test/python/test_getvalue.py
src/test/python/test_invoke.py
src/test/python/test_iterators.py
src/test/python/test_numpy.py
src/test/python/test_preinits.py
src/test/python/test_shared_interpreter.py
src/test/python/test_shared_modules.py
src/test/python/test_synchronized.py

At least, we could disable all those tests as well in Fedora. But i don't like that idea because all those tests are randomized across all the test files mixed with native python. In my opinion, some tests with java code should be separately called with a java process instead as they're written in java what fails here by calls to subprocess Popen. Upstream decided for unknown reasons to call all tests from python, propably to have them all accessible with python unittest.

$ grep -l jep_pipe src/test/python/*.py
src/test/python/jep_pipe.py
src/test/python/test_exceptions.py
src/test/python/test_getvalue.py
src/test/python/test_invoke.py
src/test/python/test_iterators.py
src/test/python/test_numpy.py
src/test/python/test_preinits.py
src/test/python/test_run_script.py
src/test/python/test_shared_interpreter.py
src/test/python/test_shared_modules.py
src/test/python/test_synchronized.py

Comment 6 Raphael Groner 2019-05-12 14:01:58 UTC
Obviously, there's need to work with upstream to get a good patch. Any help is appreciated.

JPype is the preferred way how to call java from inside python. There's JPackage to import (custom) classes and call methods, but the official documentation is stingy about how to do it for custom classes like tests are.

https://jpype.readthedocs.io/en/latest/userguide.html#jpackage-class

Comment 7 Raphael Groner 2019-05-12 15:50:30 UTC
FTR Besides JPype (mostly recommended in official discussions), there's also Py4J. Though, Py4J needs some kind of wrapper with so called entry points to import any java classes. 
https://www.py4j.org/getting_started.html#writing-the-java-program

Third option: Use pyjnius. But I fail as well there to get any working sample with the native java tests provided in python-jep.
https://pyjnius.readthedocs.io/en/stable/quickstart.html#a-minimal-example

That proposed wrapper logic means, we'd have to write even more java code to fix our initial issue with Popen and prevent any programatically forbidden forks as explained, to properly execute any java inside the main thread of initial python interpreter.

Comment 8 Raphael Groner 2019-05-24 10:31:04 UTC
Another option is to also look deeper into os.exec*().
https://bugs.python.org/issue34651#msg343364
https://www.python-kurs.eu/forking.php

Comment 9 Charalampos Stratakis 2019-07-23 14:32:48 UTC
Changing to assigned, as verified means that a fix is actually verified.

Comment 10 Raphael Groner 2019-07-29 09:01:18 UTC
No glue how to easily fix in near future. Let's hope for upstream.

Comment 11 Ben Cotton 2019-08-13 16:51:48 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 12 Ben Cotton 2019-08-13 18:53:26 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 13 Miro Hrončok 2019-08-14 22:24:19 UTC
The coordinated rebuild of Python 3.8 has started in the `f32-python` side tag.

If you figure out how to rebuild this package, please don't rebuild it in regular rawhide, but use the side tag instead:

    on branch master:
    $ fedpkg build --target=f32-python

To wait for a build to show up in the side tag, do:

    $ koji wait-repo f32-python --build=<nvr>

Where <nvr> is name-version-release of the source package, e.g. python-foo-1.1-2.fc32.

An updated mock config is posted at:
http://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Note that it will take a while before the essential packages are rebuilt, so don't expect all your dependencies to be available right away.

Thanks. Let us know if you need up to date info, or if you have any questions.



PS this message is mass posted to all the bugs that block the PYTHON38 bug. If this is also a Fedora 31 FTBFS bug and you manage to fix it, you can do a f31 build as usual:

    on branch f31:
    $ fedpkg build

Comment 14 Miro Hrončok 2019-08-21 16:34:33 UTC
The f32-python side tag has been merged. In order to rebuild the package, do it in regular rawhide, but please wait until python3-3.8 is tagged:

  $ koji wait-repo f32-build --build python3-3.8.0~b3-3.fc32


If your built already started in f32-python, after it is finished, please tag it to rawhide with:

  $ koji tag-build f32-pending <nvr>

For example:

  $ koji tag-build f32-pending libreoffice-6.3.0.4-3.fc32

Thanks!

(This comment is mass posted to all bugzillas blocking the PYTHON38 tracking bug.)

Comment 15 Miro Hrončok 2019-08-21 17:29:07 UTC
(Python 3.8 has landed in the rawhide buildroot.)

Comment 16 Raphael Groner 2019-08-31 15:34:58 UTC
Fixed in rawhide with new upstream release. All tests pass successfully. For more information, see changelog provided by upstream.


Note You need to log in before you can comment on or make changes to this bug.