Bug 1595421 - fontforge on Python 3.7: Fatal Python error: _Py_InitializeCore: main interpreter already initialized
Summary: fontforge on Python 3.7: Fatal Python error: _Py_InitializeCore: main interpr...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
Assignee: Victor Stinner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON37 1604723 1604724 1604726 1604728 1604729 1604730 1604731 1604732 1604733 1604734
TreeView+ depends on / blocked
 
Reported: 2018-06-26 21:50 UTC by Miro Hrončok
Modified: 2018-07-23 12:33 UTC (History)
17 users (show)

Fixed In Version: python3-3.7.0-4.fc29, fontforge-20170731-10.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-23 12:29:25 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Python 34008 None None None 2018-06-30 10:02:00 UTC

Description Miro Hrončok 2018-06-26 21:50:44 UTC
Description of problem: When other packages try to use fontforge in the f29-python side tag (with Python 3.7), it fails:

fontforge -lang=py -script apply_featurefile.py Lohit-Assamese.sfd Lohit-Assamese.fea
Copyright (c) 2000-2014 by George Williams. See AUTHORS for Contributors.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 with many parts BSD <http://fontforge.org/license.html>. Please read LICENSE.
 Based on sources from 08:52 UTC 19-Jun-2018-ML-D.
 Based on source from git with hash: 
Fatal Python error: _Py_InitializeCore: main interpreter already initialized


Version-Release number of selected component (if applicable): fontforge-20170731-6.fc29


_Py_InitializeCore can only be called once. See https://bugs.python.org/issue33932 or similar change in https://github.com/originell/jpype/pull/332

Even if this change might get reverted in 3.7, this will need to be fixed before 3.8 anyway.

Comment 1 Miro Hrončok 2018-06-26 21:52:15 UTC
Victor, should I backport https://github.com/python/cpython/commit/3747dd16d5d2af3499f586386e49740a0454cf44 to our python3 package?

Comment 2 Miro Hrončok 2018-06-26 21:53:52 UTC
> _Py_InitializeCore can only be called once

Should have been:

> Py_Initialize() can only be called once

Comment 3 Miro Hrončok 2018-06-27 09:19:46 UTC
Ok, according to the release manager of 3.7, the fix will be in final 3.7.0, I'll backport it to unblock this.

Comment 4 Miro Hrončok 2018-06-27 09:20:52 UTC
Actually, 3.7.0 final is planned for today, so I'll just wait.

Comment 5 Parag Nemade 2018-06-27 14:31:24 UTC
Thank you Miro for looking into this issue.

Comment 6 Miro Hrončok 2018-06-28 09:49:29 UTC
python3-3.7.0-1.fc29 built. will rebuild fontforge.

Comment 7 Miro Hrončok 2018-06-28 10:15:44 UTC
Built. Will try to build lohit-*-fonts to see if it's fixed.

Comment 8 Miro Hrončok 2018-06-28 10:25:58 UTC
fontforge -lang=py -script apply_featurefile.py Lohit-Assamese.sfd Lohit-Assamese.fea
Copyright (c) 2000-2014 by George Williams. See AUTHORS for Contributors.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 with many parts BSD <http://fontforge.org/license.html>. Please read LICENSE.
 Based on sources from 09:57 UTC 28-Jun-2018-ML-D.
 Based on source from git with hash: 
Fatal Python error: _Py_InitializeCore: main interpreter already initialized

OK, Victor, what now?

Comment 9 Miro Hrončok 2018-06-28 10:56:34 UTC
#0  0x00007ffff192a14b in raise () from /lib64/libc.so.6
#1  0x00007ffff19146d1 in abort () from /lib64/libc.so.6
#2  0x00007ffff61e4c6a in fatal_error.lto_priv () from /lib64/libpython3.7m.so.1.0
#3  0x00007ffff62420a4 in _Py_FatalInitError () from /lib64/libpython3.7m.so.1.0
#4  0x00007ffff6245b5f in pymain_main.constprop.357.cold () from /lib64/libpython3.7m.so.1.0
#5  0x00007ffff6245dfe in Py_Main () from /lib64/libpython3.7m.so.1.0
#6  0x00007ffff68bb063 in PyFF_Main () from /lib64/libfontforge.so.2
#7  0x00007ffff68deb4d in CheckIsScript () from /lib64/libfontforge.so.2
#8  0x00007ffff7a6ae62 in fontforge_main () from /lib64/libfontforgeexe.so.2
#9  0x00007ffff19162fb in __libc_start_main () from /lib64/libc.so.6
#10 0x000055555555496a in _start ()

Comment 10 Victor Stinner 2018-06-28 12:42:48 UTC
Oh no. My change https://bugs.python.org/issue33932 didn't fix fontforge :-( fontforge calls Py_Initialize() and then Py_Main(). It seems like my fix doesn't cover this use case. I didn't expect that an application embeds Python using Py_Main(). For me, Py_Main() was only used by Python itself.

I reopened the issue upstream: https://bugs.python.org/issue33932#msg320665

For fontforge, the workaround/fix is to not call Py_Initialize() before Py_Main().

Comment 11 Miro Hrončok 2018-06-28 17:10:15 UTC
Victor, would you be able to provide a workaround patch for fontforge? Naively removing Py_Initialize doesn't solve this.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff629a9f5 in PyTuple_New () from /lib64/libpython3.7m.so.1.0
(gdb) backtrace
#0  0x00007ffff629a9f5 in PyTuple_New () from /lib64/libpython3.7m.so.1.0
#1  0x00007ffff6279985 in PyType_Ready () from /lib64/libpython3.7m.so.1.0
#2  0x00007ffff62799b5 in PyType_Ready () from /lib64/libpython3.7m.so.1.0
#3  0x00007ffff68b5115 in CreatePyModule () from /lib64/libfontforge.so.2
#4  0x00007ffff68baa22 in FontForge_InitializeEmbeddedPython () from /lib64/libfontforge.so.2
#5  0x00007ffff68baf5c in PyFF_Main () from /lib64/libfontforge.so.2
#6  0x00007ffff68deafd in CheckIsScript () from /lib64/libfontforge.so.2
#7  0x00007ffff7a6ae62 in fontforge_main () from /lib64/libfontforgeexe.so.2
#8  0x00007ffff19162fb in __libc_start_main () from /lib64/libc.so.6
#9  0x000055555555496a in _start ()

Comment 12 Nick Coghlan 2018-06-30 07:42:06 UTC
As an interim workaround, I'd suggest trying a CPython patch that changes the following lines to return Py_INIT_OK() instead of reporting an error:

* https://github.com/python/cpython/blob/5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L619
* https://github.com/python/cpython/blob/5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L797

I'm not sure it'll work, but it's worth a try to see if it's enough to emulate the old "It's like running Py_Main(), but without actually configuring the interpreter from the environment and command line properly" behaviour.

Comment 13 Nick Coghlan 2018-06-30 07:45:41 UTC
The relevant upstream issue has moved: https://bugs.python.org/issue34008

This is due to the fact that Py_Main() doesn't call Py_Initialize() internally any more (which is why the fix for https://bugs.python.org/issue33932 didn't fix the fontforge failure after all).

Instead, both Py_Main() and Py_Initialize() call the new lower level interpreter bootstrapping API directly.

Comment 14 Miro Hrončok 2018-06-30 11:00:22 UTC
(In reply to Nick Coghlan from comment #12)
> As an interim workaround, I'd suggest trying a CPython patch that changes
> the following lines to return Py_INIT_OK() instead of reporting an error:
> 
> *
> https://github.com/python/cpython/blob/
> 5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L619
> *
> https://github.com/python/cpython/blob/
> 5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L797
> 
> I'm not sure it'll work, but it's worth a try to see if it's enough to
> emulate the old "It's like running Py_Main(), but without actually
> configuring the interpreter from the environment and command line properly"
> behaviour.

https://koji.fedoraproject.org/koji/taskinfo?taskID=27963735 in progress (I haven't disabled the tests to see if the test suite still works).

Comment 15 Miro Hrončok 2018-06-30 12:25:35 UTC
Fatal Python error: _Py_InitializeMainInterpreter: main interpreter already initialized

Current thread 0x00007ffff7fd1e80 (most recent call first):

Program received signal SIGABRT, Aborted.
0x00007ffff192809b in raise () from /lib64/libc.so.6
(gdb) backtrace
#0  0x00007ffff192809b in raise () at /lib64/libc.so.6
#1  0x00007ffff19126f9 in abort () at /lib64/libc.so.6
#2  0x00007ffff61e3dac in fatal_error.lto_priv () at /lib64/libpython3.7m.so.1.0
#3  0x00007ffff6251035 in pathconfig_global_init.cold () at /lib64/libpython3.7m.so.1.0
#4  0x00007ffff6254e4d in pymain_main.constprop.361.cold () at /lib64/libpython3.7m.so.1.0
#5  0x00007ffff62558db in Py_Main () at /lib64/libpython3.7m.so.1.0
#6  0x00007ffff68bb063 in PyFF_Main () at /lib64/libfontforge.so.2
#7  0x00007ffff68deb4d in CheckIsScript () at /lib64/libfontforge.so.2
#8  0x00007ffff7a6ae62 in fontforge_main () at /lib64/libfontforgeexe.so.2
#9  0x00007ffff191425b in __libc_start_main () at /lib64/libc.so.6
#10 0x000055555555496a in _start ()

Comment 16 Miro Hrončok 2018-07-02 18:20:31 UTC
The side tag is no more. This now affects rawhide.

Comment 17 Parag Nemade 2018-07-05 01:05:53 UTC
It appears fontforge got built successfully now. As part of other bug 1596037 fix, Kevin attempted new fontforge build and its successful.

Comment 18 Miro Hrončok 2018-07-05 08:25:07 UTC
Fontforge builds from the start. It's:

lohit-assamese-fonts
lohit-bengali-fonts
lohit-devanagari-fonts
lohit-gujarati-fonts
lohit-gurmukhi-fonts
lohit-malayalam-fonts
lohit-marathi-fonts
lohit-nepali-fonts
lohit-odia-fonts
lohit-tamil-fonts

that FTBFS due this bug.

Comment 19 Parag Nemade 2018-07-05 09:04:17 UTC
Ah right, those fonts packages are FTBFS now.

Comment 20 Victor Stinner 2018-07-12 15:35:52 UTC
I looked one more time to the issue:

* fontforge code is fine: it calls Py_Initialize() before using the Python C API to initialize its Python namespace (modules, functions, types, etc.), and later it calls Py_Main().

* Python 3.7.0 is wrong: it fails with a fatal erro when Py_Main() is called if Py_Initialize() has been called previously. It is a regression since it worked fine in Python 3.6.

* My PR https://github.com/python/cpython/pull/8043 works again the simplest test: "fontforge -script hello.py" where hello.py is just "print('Hello World!')"

* This PR is not enough according to Miro, since fontforge requires sys.argv to be set, whereas sys.argv is no longer set when Py_Main() is after Py_Initialize(): https://github.com/python/cpython/pull/8043#issuecomment-401731992

Comment 21 Parag Nemade 2018-07-18 09:28:40 UTC
Any updates here please? I see those fonts are still FTBFS.

Comment 22 Victor Stinner 2018-07-20 15:40:11 UTC
> Any updates here please? I see those fonts are still FTBFS.

I updated my upstream PR https://github.com/python/cpython/pull/8043 to not only allow to call Py_Main() after Py_Initialize(), but also to update sys.argv from the new configuration.

Miro confirmed that my PR fixes fontforge:
https://github.com/python/cpython/pull/8043#issuecomment-406629309

I just merged my PR to master and I'm backporting it to Python 3.7:
https://github.com/python/cpython/pull/8352

Sorry for the delay, Python initialization code is very complex and so very fragile. I hesitated how to fix the ssue. I chose to write the smallest fix just to repair fontforge. More enhancements will only be done in the future Python 3.8.


Note You need to log in before you can comment on or make changes to this bug.