Bug 1595421

Victor, should I backport https://github.com/python/cpython/commit/3747dd16d5d2af3499f586386e49740a0454cf44 to our python3 package?

> _Py_InitializeCore can only be called once

Should have been:

> Py_Initialize() can only be called once

Ok, according to the release manager of 3.7, the fix will be in final 3.7.0, I'll backport it to unblock this.

Actually, 3.7.0 final is planned for today, so I'll just wait.

Thank you Miro for looking into this issue.

python3-3.7.0-1.fc29 built. will rebuild fontforge.

Built. Will try to build lohit-*-fonts to see if it's fixed.

fontforge -lang=py -script apply_featurefile.py Lohit-Assamese.sfd Lohit-Assamese.fea
Copyright (c) 2000-2014 by George Williams. See AUTHORS for Contributors.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 with many parts BSD <http://fontforge.org/license.html>. Please read LICENSE.
 Based on sources from 09:57 UTC 28-Jun-2018-ML-D.
 Based on source from git with hash: 
Fatal Python error: _Py_InitializeCore: main interpreter already initialized

OK, Victor, what now?

#0  0x00007ffff192a14b in raise () from /lib64/libc.so.6
#1  0x00007ffff19146d1 in abort () from /lib64/libc.so.6
#2  0x00007ffff61e4c6a in fatal_error.lto_priv () from /lib64/libpython3.7m.so.1.0
#3  0x00007ffff62420a4 in _Py_FatalInitError () from /lib64/libpython3.7m.so.1.0
#4  0x00007ffff6245b5f in pymain_main.constprop.357.cold () from /lib64/libpython3.7m.so.1.0
#5  0x00007ffff6245dfe in Py_Main () from /lib64/libpython3.7m.so.1.0
#6  0x00007ffff68bb063 in PyFF_Main () from /lib64/libfontforge.so.2
#7  0x00007ffff68deb4d in CheckIsScript () from /lib64/libfontforge.so.2
#8  0x00007ffff7a6ae62 in fontforge_main () from /lib64/libfontforgeexe.so.2
#9  0x00007ffff19162fb in __libc_start_main () from /lib64/libc.so.6
#10 0x000055555555496a in _start ()

Oh no. My change https://bugs.python.org/issue33932 didn't fix fontforge :-( fontforge calls Py_Initialize() and then Py_Main(). It seems like my fix doesn't cover this use case. I didn't expect that an application embeds Python using Py_Main(). For me, Py_Main() was only used by Python itself.

I reopened the issue upstream: https://bugs.python.org/issue33932#msg320665

For fontforge, the workaround/fix is to not call Py_Initialize() before Py_Main().

Victor, would you be able to provide a workaround patch for fontforge? Naively removing Py_Initialize doesn't solve this.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff629a9f5 in PyTuple_New () from /lib64/libpython3.7m.so.1.0
(gdb) backtrace
#0  0x00007ffff629a9f5 in PyTuple_New () from /lib64/libpython3.7m.so.1.0
#1  0x00007ffff6279985 in PyType_Ready () from /lib64/libpython3.7m.so.1.0
#2  0x00007ffff62799b5 in PyType_Ready () from /lib64/libpython3.7m.so.1.0
#3  0x00007ffff68b5115 in CreatePyModule () from /lib64/libfontforge.so.2
#4  0x00007ffff68baa22 in FontForge_InitializeEmbeddedPython () from /lib64/libfontforge.so.2
#5  0x00007ffff68baf5c in PyFF_Main () from /lib64/libfontforge.so.2
#6  0x00007ffff68deafd in CheckIsScript () from /lib64/libfontforge.so.2
#7  0x00007ffff7a6ae62 in fontforge_main () from /lib64/libfontforgeexe.so.2
#8  0x00007ffff19162fb in __libc_start_main () from /lib64/libc.so.6
#9  0x000055555555496a in _start ()

As an interim workaround, I'd suggest trying a CPython patch that changes the following lines to return Py_INIT_OK() instead of reporting an error:

* https://github.com/python/cpython/blob/5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L619
* https://github.com/python/cpython/blob/5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L797

I'm not sure it'll work, but it's worth a try to see if it's enough to emulate the old "It's like running Py_Main(), but without actually configuring the interpreter from the environment and command line properly" behaviour.

The relevant upstream issue has moved: https://bugs.python.org/issue34008

This is due to the fact that Py_Main() doesn't call Py_Initialize() internally any more (which is why the fix for https://bugs.python.org/issue33932 didn't fix the fontforge failure after all).

Instead, both Py_Main() and Py_Initialize() call the new lower level interpreter bootstrapping API directly.

(In reply to Nick Coghlan from comment #12)
> As an interim workaround, I'd suggest trying a CPython patch that changes
> the following lines to return Py_INIT_OK() instead of reporting an error:
> 
> *
> https://github.com/python/cpython/blob/
> 5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L619
> *
> https://github.com/python/cpython/blob/
> 5bb5bbfca847524bab5f2368bdb48eedf5dba74f/Python/pylifecycle.c#L797
> 
> I'm not sure it'll work, but it's worth a try to see if it's enough to
> emulate the old "It's like running Py_Main(), but without actually
> configuring the interpreter from the environment and command line properly"
> behaviour.

https://koji.fedoraproject.org/koji/taskinfo?taskID=27963735 in progress (I haven't disabled the tests to see if the test suite still works).

Fatal Python error: _Py_InitializeMainInterpreter: main interpreter already initialized

Current thread 0x00007ffff7fd1e80 (most recent call first):

Program received signal SIGABRT, Aborted.
0x00007ffff192809b in raise () from /lib64/libc.so.6
(gdb) backtrace
#0  0x00007ffff192809b in raise () at /lib64/libc.so.6
#1  0x00007ffff19126f9 in abort () at /lib64/libc.so.6
#2  0x00007ffff61e3dac in fatal_error.lto_priv () at /lib64/libpython3.7m.so.1.0
#3  0x00007ffff6251035 in pathconfig_global_init.cold () at /lib64/libpython3.7m.so.1.0
#4  0x00007ffff6254e4d in pymain_main.constprop.361.cold () at /lib64/libpython3.7m.so.1.0
#5  0x00007ffff62558db in Py_Main () at /lib64/libpython3.7m.so.1.0
#6  0x00007ffff68bb063 in PyFF_Main () at /lib64/libfontforge.so.2
#7  0x00007ffff68deb4d in CheckIsScript () at /lib64/libfontforge.so.2
#8  0x00007ffff7a6ae62 in fontforge_main () at /lib64/libfontforgeexe.so.2
#9  0x00007ffff191425b in __libc_start_main () at /lib64/libc.so.6
#10 0x000055555555496a in _start ()

The side tag is no more. This now affects rawhide.

It appears fontforge got built successfully now. As part of other bug 1596037 fix, Kevin attempted new fontforge build and its successful.

Fontforge builds from the start. It's:

lohit-assamese-fonts
lohit-bengali-fonts
lohit-devanagari-fonts
lohit-gujarati-fonts
lohit-gurmukhi-fonts
lohit-malayalam-fonts
lohit-marathi-fonts
lohit-nepali-fonts
lohit-odia-fonts
lohit-tamil-fonts

that FTBFS due this bug.

Ah right, those fonts packages are FTBFS now.

I looked one more time to the issue:

* fontforge code is fine: it calls Py_Initialize() before using the Python C API to initialize its Python namespace (modules, functions, types, etc.), and later it calls Py_Main().

* Python 3.7.0 is wrong: it fails with a fatal erro when Py_Main() is called if Py_Initialize() has been called previously. It is a regression since it worked fine in Python 3.6.

* My PR https://github.com/python/cpython/pull/8043 works again the simplest test: "fontforge -script hello.py" where hello.py is just "print('Hello World!')"

* This PR is not enough according to Miro, since fontforge requires sys.argv to be set, whereas sys.argv is no longer set when Py_Main() is after Py_Initialize(): https://github.com/python/cpython/pull/8043#issuecomment-401731992

Any updates here please? I see those fonts are still FTBFS.

> Any updates here please? I see those fonts are still FTBFS.

I updated my upstream PR https://github.com/python/cpython/pull/8043 to not only allow to call Py_Main() after Py_Initialize(), but also to update sys.argv from the new configuration.

Miro confirmed that my PR fixes fontforge:
https://github.com/python/cpython/pull/8043#issuecomment-406629309

I just merged my PR to master and I'm backporting it to Python 3.7:
https://github.com/python/cpython/pull/8352

Sorry for the delay, Python initialization code is very complex and so very fragile. I hesitated how to fix the ssue. I chose to write the smallest fix just to repair fontforge. More enhancements will only be done in the future Python 3.8.

https://src.fedoraproject.org/rpms/python3/pull-request/46