Bug 1743896

Summary: paraview fails to build with Python 3.8
Product: [Fedora] Fedora Reporter: Miro Hrončok <mhroncok>
Component: paraviewAssignee: Miro Hrončok <mhroncok>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: anto.trande, dakingun, lbalhar, mplch, orion, pviktori, vstinner
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: paraview-5.6.0-9.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-06 01:19:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1744292    
Bug Blocks: 1686977    

Description Miro Hrončok 2019-08-20 23:14:07 UTC
paraview-5.6.0-8.fc32 failed to rebuild with Python 3.8.

See https://koji.fedoraproject.org/koji/buildinfo?buildID=1358771

The error is:

BUILDSTDERR: /builddir/build/BUILD/ParaView-v5.6.0/VTK/Wrapping/PythonCore/PyVTKNamespace.cxx:93:1: error: cannot convert 'std::nullptr_t' to 'Py_ssize_t' {aka 'long int'} in initialization

In Python 3.8, the reserved "tp_print" slot was changed from a function pointer to a number, `Py_ssize_t tp_vectorcall_offset`.
In C, there is no "nullptr"; either a 0 or NULL casts automatically to both pointers and numbers.

Please use 0 instead of "nullptr" in the slot to be source-compatible both with Python 3.8 and previous versions.

See this as an example PR: https://github.com/YafaRay/Core/pull/114




This issue is blocking the Python 3.8 rebuilds. If this package won't build with 3.8, it won't be installable, along with all its dependent packages, after the side tag is merged.
Furthermore, as it fails to install, its dependent packages will fail to install and/or build as well. 


The coordinated rebuild of Python 3.8 has started in the `f32-python` side tag.

If you figure out how to rebuild this package, please don't rebuild it in regular rawhide, but use the side tag instead:

    on branch master:
    $ fedpkg build --target=f32-python

To wait for a build to show up in the side tag, do:

    $ koji wait-repo f32-python --build=<nvr>

Where <nvr> is name-version-release of the source package, e.g. python-foo-1.1-2.fc32.

An updated mock config is posted at: http://copr.fedorainfracloud.org/coprs/g/python/python3.8/

Thanks. Let us know if you need up to date info, or if you have any questions.

Comment 1 Miro Hrončok 2019-08-21 16:35:07 UTC
The f32-python side tag has been merged. In order to rebuild the package, do it in regular rawhide, but please wait until python3-3.8 is tagged:

  $ koji wait-repo f32-build --build python3-3.8.0~b3-3.fc32


If your built already started in f32-python, after it is finished, please tag it to rawhide with:

  $ koji tag-build f32-pending <nvr>

For example:

  $ koji tag-build f32-pending libreoffice-6.3.0.4-3.fc32

Thanks!

(This comment is mass posted to all bugzillas blocking the PYTHON38 tracking bug.)

Comment 2 Miro Hrončok 2019-08-21 17:29:12 UTC
(Python 3.8 has landed in the rawhide buildroot.)

Comment 3 Orion Poplawski 2019-08-23 03:40:01 UTC
make[3]: Entering directory '/builddir/build/BUILD/ParaView-v5.6.0/x86_64-redhat-linux-gnu'
BUILDSTDERR: cd /builddir/build/BUILD/ParaView-v5.6.0/x86_64-redhat-linux-gnu && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /builddir/build/BUILD/ParaView-v5.6.0 /builddir/build/BUILD/ParaView-v5.6.0/CommandLineExecutables /builddir/build/BUILD/ParaView-v5.6.0/x86_64-redhat-linux-gnu /builddir/build/BUILD/ParaView-v5.6.0/x86_64-redhat-linux-gnu/CommandLineExecutables /builddir/bFatal Python error: no mem for sys.argv
BUILDSTDERR: SystemError: /builddir/build/BUILD/Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function
BUILDSTDERR: Current thread 0x00007f6f456c7900 (most recent call first):
BUILDSTDERR: make[3]: *** [Utilities/Sphinx/CMakeFiles/ParaViewPythonDoc.dir/build.make:60: Utilities/Sphinx/CMakeFiles/ParaViewPythonDoc] Aborted (core dumped)
BUILDSTDERR: make[2]: *** [CMakeFiles/Makefile2:52830: Utilities/Sphinx/CMakeFiles/ParaViewPythonDoc.dir/all] Error 2
BUILDSTDERR: make[1]: *** [CMakeFiles/Makefile2:162: CMakeFiles/ParaViewDoc.dir/rule] Error 2
BUILDSTDERR: make: *** [Makefile:215: ParaViewDoc] Error 2
BUILDSTDERR: error: Bad exit status from /var/tmp/rpm-tmp.uW7bXl (%build)
uild/BUILD/ParaView-v5.6.0/x86_64-redhat-linux-gnu/CommandLineExecutables/CMakeFiles/pvpython.dir/DependInfo.cmake --color=

unfortunately the stderr and stdout is mixed, but it seems like we're hitting some kind of python memory issue.

Comment 4 Miro Hrončok 2019-08-23 05:13:08 UTC
Hmm... there are several dangerous lines:


error: no mem for sys.argv

BUILDSTDERR: SystemError: /builddir/build/BUILD/Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function

Aborted (core dumped)

Comment 5 Miro Hrončok 2019-08-23 05:20:20 UTC
Python-3.8.0b3/Objects/unicodeobject.c:2089:

PyObject *
PyUnicode_FromWideChar(const wchar_t *u, Py_ssize_t size)
{
    PyObject *unicode;
    Py_UCS4 maxchar = 0;
    Py_ssize_t num_surrogates;

    if (u == NULL && size != 0) {
        PyErr_BadInternalCall();               <-------- 2089
        return NULL;
    }



ParaView-v5.6.0/VTK/Utilities/PythonInterpreter/vtkPythonInterpreter.cxx has:

    // setup default argv. Without this, code snippets that check `sys.argv` may
    // fail when run in embedded VTK Python environment.
    PySys_SetArgvEx(0, nullptr, 0);


I wonder whether PySys_SetArgvEx isn't broken for 0.

Comment 6 Miro Hrončok 2019-08-23 05:54:50 UTC
#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PySys_SetArgvEx(0, NULL, 0);
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    PyMem_RawFree(program);
    return 0;
}


$ gcc $(python3.7-config --cflags --ldflags) example.c 
$ ./a.out 
Today is Fri Aug 23 07:54:17 2019
$ gcc $(python3.8-config --cflags --ldflags --embed) example.c 
$ ./a.out 
Fatal Python error: no mem for sys.argv
SystemError: /builddir/build/BUILD/Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function

Current thread 0x00007f05f6328740 (most recent call first):
Aborted (core dumped)

Will report this as a regression.

Comment 7 Miro Hrončok 2019-08-23 09:39:28 UTC
A Python fix is at https://github.com/python/cpython/pull/15415

I suggest we don't backport it but instead wait for the next beta scheduled for Monday (2019-08-26), assuming the fix will be included there.