Bug 1965584

This seems to be python or possibly python-gevent crashing when running sphinx-build-3.  Running with gdb just results in a corrupted stack trace.  valgrind reports:

[mockbuild@d4b708504a514540a8c1ae53566cb8a6 ~]$ cd build/BUILD/python-x2go-0.6.1.3/docs
[mockbuild@d4b708504a514540a8c1ae53566cb8a6 docs]$ valgrind /usr/bin/sphinx-build-3 -b html -d build/doctrees   source build/html
==35== Memcheck, a memory error detector
==35== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==35== Using Valgrind-3.17.0 and LibVEX; rerun with -h for copyright info
==35== Command: /usr/bin/sphinx-build-3 -b html -d build/doctrees source build/html
==35== 
Running Sphinx v3.5.4
==84== Warning: invalid file descriptor 1024 in syscall close()
==84== Warning: invalid file descriptor 1025 in syscall close()
==84== Warning: invalid file descriptor 1026 in syscall close()
==84== Warning: invalid file descriptor 1027 in syscall close()
==84==    Use --log-fd=<number> to select an alternative log fd.
==84== Warning: invalid file descriptor 1028 in syscall close()
==84== Warning: invalid file descriptor 1029 in syscall close()
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
loading translations [en]... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 46 source files that are out of date
updating environment: [new config] 46 added, 0 changed, 0 removed
reading sources... [  2%] index
reading sources... [  4%] modules
reading sources... [  6%] x2go
reading sources... [  8%] x2go.backends
reading sources... [ 10%] x2go.backends.control
reading sources... [ 13%] x2go.backends.control.plain
reading sources... [ 15%] x2go.backends.info
reading sources... [ 17%] x2go.backends.info.plain
reading sources... [ 19%] x2go.backends.printing
reading sources... [ 21%] x2go.backends.printing.file
reading sources... [ 23%] x2go.backends.profiles
reading sources... [ 26%] x2go.backends.profiles.base
reading sources... [ 28%] x2go.backends.profiles.file
reading sources... [ 30%] x2go.backends.profiles.httpbroker
reading sources... [ 32%] x2go.backends.profiles.sshbroker
reading sources... [ 34%] x2go.backends.proxy
reading sources... [ 36%] x2go.backends.proxy.base
reading sources... [ 39%] x2go.backends.proxy.nx3
reading sources... [ 41%] x2go.backends.settings
reading sources... [ 43%] x2go.backends.settings.file
reading sources... [ 45%] x2go.backends.terminal
reading sources... [ 47%] x2go.backends.terminal.plain
==91== Warning: invalid file descriptor 1024 in syscall close()
==91== Warning: invalid file descriptor 1025 in syscall close()
==91== Warning: invalid file descriptor 1026 in syscall close()
==91== Warning: invalid file descriptor 1027 in syscall close()
==91==    Use --log-fd=<number> to select an alternative log fd.
==91== Warning: invalid file descriptor 1028 in syscall close()
==91== Warning: invalid file descriptor 1029 in syscall close()
==91== 
==91== HEAP SUMMARY:
==91==     in use at exit: 56,452,639 bytes in 499,640 blocks
==91==   total heap usage: 7,604,380 allocs, 7,104,740 frees, 991,297,424 bytes allocated
==91== 
==35== Invalid read of size 8
==35==    at 0xA87926F: __pyx_f_6gevent_24_gevent_c_hub_primitives__primitive_wait.lto_priv.0 (_hub_primitives.c:6044)
==35==    by 0xA87AE57: __pyx_f_6gevent_24_gevent_c_hub_primitives_wait_on_watcher.lto_priv.0 (_hub_primitives.c:6661)
==35==    by 0xA87AFA8: UnknownInlinedFun (_hub_primitives.c:6801)
==35==    by 0xA87AFA8: __pyx_pw_6gevent_24_gevent_c_hub_primitives_9wait_on_watcher.lto_priv.0 (_hub_primitives.c:6776)
==35==    by 0x49627F3: _PyObject_MakeTpCall (call.c:215)
==35==    by 0x495ED5D: UnknownInlinedFun (abstract.h:112)
==35==    by 0x495ED5D: UnknownInlinedFun (abstract.h:99)
==35==    by 0x495ED5D: UnknownInlinedFun (abstract.h:123)
==35==    by 0x495ED5D: UnknownInlinedFun (ceval.c:5885)
==35==    by 0x495ED5D: _PyEval_EvalFrameDefault (ceval.c:4214)
==35==    by 0x4958943: UnknownInlinedFun (pycore_ceval.h:46)
==35==    by 0x4958943: _PyEval_Vector (ceval.c:5069)
==35==    by 0x4959D0D: UnknownInlinedFun (abstract.h:114)
==35==    by 0x4959D0D: UnknownInlinedFun (abstract.h:123)
==35==    by 0x4959D0D: UnknownInlinedFun (ceval.c:5885)
==35==    by 0x4959D0D: _PyEval_EvalFrameDefault (ceval.c:4199)
==35==    by 0x4958943: UnknownInlinedFun (pycore_ceval.h:46)
==35==    by 0x4958943: _PyEval_Vector (ceval.c:5069)
==35==    by 0x496F192: UnknownInlinedFun (call.c:342)
==35==    by 0x496F192: UnknownInlinedFun (abstract.h:114)
==35==    by 0x496F192: method_vectorcall (classobject.c:61)
==35==    by 0x49C410B: UnknownInlinedFun (abstract.h:114)
==35==    by 0x49C410B: _PyObject_CallNoArg.lto_priv.0 (abstract.h:168)
==35==    by 0x49E10BF: UnknownInlinedFun (bufferedio.c:1551)
==35==    by 0x49E10BF: UnknownInlinedFun (bufferedio.c:888)
==35==    by 0x49E10BF: _io__Buffered_read (bufferedio.c.h:169)
==35==    by 0x496728F: cfunction_vectorcall_FASTCALL (methodobject.c:426)
==35==  Address 0x1ffeff8298 is on thread 1's stack
==35==  1656 bytes below stack pointer
==35== 
==35== Invalid write of size 8
==35==    at 0x4948D59: _PyObject_GC_Alloc (gcmodule.c:2257)
==35==  Address 0x1ffeff8288 is on thread 1's stack
==35==  1664 bytes below stack pointer
==35== 
==35== Use of uninitialised value of size 8
==35==    at 0x4948D71: _PyObject_GC_Alloc (gcmodule.c:2272)
==35==    by 0xA70FB2F: ???
==35== 
==35== Jump to the invalid address stated on the next line
==35==    at 0x0: ???
==35==    by 0xA70FB2F: ???
==35==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==35== 
==35== 
==35== Process terminating with default action of signal 11 (SIGSEGV)
==35==  Bad permissions for mapped region at address 0x0
==35==    at 0x0: ???
==35==    by 0xA70FB2F: ???

reading sources... [ 47%] x2go.backends.terminal.plain
make: *** [Makefile:53: html] Segmentation fault (core dumped)

It would be nice to retrieve the stack trace here.

==35== Invalid read of size 8
==35==    at 0xA87926F: __pyx_f_6gevent_24_gevent_c_hub_primitives__primitive_wait.lto_priv.0 (_hub_primitives.c:6044)
==35==    by 0xA87AE57: __pyx_f_6gevent_24_gevent_c_hub_primitives_wait_on_watcher.lto_priv.0 (_hub_primitives.c:6661)
==35==    by 0xA87AFA8: UnknownInlinedFun (_hub_primitives.c:6801)
==35==    by 0xA87AFA8: __pyx_pw_6gevent_24_gevent_c_hub_primitives_9wait_on_watcher.lto_priv.0 (_hub_primitives.c:6776)

These 4 functions come from the following file of gevent compiled with Cython:
https://github.com/gevent/gevent/blob/master/src/gevent/_hub_primitives.py

The crash comes from the gevent code compiled with Cython. Reproducer code:
---
# Disable C accelerator
import gevent._compat
#gevent._compat.PURE_PYTHON = True

from gevent import _greenlet_primitives
# Explicitly import the Cython code
#from gevent import _gevent_c_greenlet_primitives as _greenlet_primitives

print(_greenlet_primitives)
SwitchOutGreenletWithLoop = _greenlet_primitives.SwitchOutGreenletWithLoop

class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): # pylint:disable=undefined-variable
    ...
---

The pure Python code of gevent works well: uncomment "PURE_PYTHON = True" line, it no longer crashes.



When using Python 3.10 built in debug mode, I get an assertion error at gevent/_hub_primitives.py, line 44:
"class WaitOperationsGreenlet(SwitchOutGreenletWithLoop):"

gevent simplified code:
---
from gevent import _greenlet_primitives

# In Cython, we define these as 'cdef inline' functions. The
# compilation unit cannot have a direct assignment to them (import
# is assignment) without generating a 'lvalue is not valid target'
# error.
locals()['SwitchOutGreenletWithLoop'] = _greenlet_primitives.SwitchOutGreenletWithLoop

class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): # pylint:disable=undefined-variable
    ...
---

_greenlet_primitives.py simplified code:
---
import greenlet

class TrackedRawGreenlet(greenlet.greenlet):
    ...

class SwitchOutGreenletWithLoop(TrackedRawGreenlet):
    ...
---


Assertion error:

python: Objects/typeobject.c :2219 : extra_ivars:  l'assertion « t_size >= b_size » a échoué.


Program received signal SIGABRT, Aborted.
0x00007ffff7c662a2 in raise () from /lib64/libc.so.6

(gdb) frame 4
#4  0x0000000000495baa in extra_ivars (type=0x7fffe78bd400 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_TrackedRawGreenlet>, base=0x7fffe9d01620 <PyGreenlet_Type>)
    at Objects/typeobject.c:2219
2219	    assert(t_size >= b_size); /* Else type smaller than base! */

(gdb) p t_size
$1 = 152
(gdb) p b_size
$2 = 160

(gdb) l
2214	extra_ivars(PyTypeObject *type, PyTypeObject *base)
2215	{
2216	    size_t t_size = type->tp_basicsize;
2217	    size_t b_size = base->tp_basicsize;
2218	
2219	    assert(t_size >= b_size); /* Else type smaller than base! */
2220	    if (type->tp_itemsize || base->tp_itemsize) {
2221	        /* If itemsize is involved, stricter rules */
2222	        return t_size != b_size ||
2223	            type->tp_itemsize != base->tp_itemsize;

(gdb) p base->tp_name
$3 = 0x7fffe9cfea2c "greenlet.greenlet"
(gdb) p type->tp_name
$4 = 0x7fffe78b8938 "gevent._gevent_c_greenlet_primitives.TrackedRawGreenlet"

(gdb) where
#0  0x00007ffff7c662a2 in raise () from /lib64/libc.so.6
#1  0x00007ffff7c4f8a4 in abort () from /lib64/libc.so.6
#2  0x00007ffff7c4f789 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff7c5ea16 in __assert_fail () from /lib64/libc.so.6
#4  0x0000000000495baa in extra_ivars (type=0x7fffe78bd400 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_TrackedRawGreenlet>, base=0x7fffe9d01620 <PyGreenlet_Type>)
    at Objects/typeobject.c:2219
#5  0x0000000000495cf5 in solid_base (type=0x7fffe78bd400 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_TrackedRawGreenlet>) at Objects/typeobject.c:2246
#6  0x0000000000495cd4 in solid_base (type=0x7fffe78bd600 <__pyx_type_6gevent_29_gevent_c_greenlet_primitives_SwitchOutGreenletWithLoop>) at Objects/typeobject.c:2243
#7  0x0000000000495aae in best_base (bases=(<type at remote 0x7fffe78bd600>,)) at Objects/typeobject.c:2189
#8  0x0000000000497fb5 in type_new_get_bases (ctx=0x7ffffffde1a0, type=0x7ffffffde198) at Objects/typeobject.c:3269
#9  0x0000000000498168 in type_new (metatype=0x893bc0 <PyType_Type>, 
    args=('WaitOperationsGreenlet', (<type at remote 0x7fffe78bd600>,), {'__module__': 'gevent._hub_primitives', '__qualname__': 'WaitOperationsGreenlet', 'wait': <function at remote 0x7fffe790b280>, 'cancel_waits_close_and_then': <function at remote 0x7fffe790b330>, '_cancel_waits_then': <function at remote 0x7fffe790b3e0>, 'cancel_wait': <function at remote 0x7fffe790b490>, '_cancel_wait': <function at remote 0x7fffe790b540>}), kwds=0x0) at Objects/typeobject.c:3311
#10 0x0000000000493518 in type_call (type=0x893bc0 <PyType_Type>, 
    args=('WaitOperationsGreenlet', (<type at remote 0x7fffe78bd600>,), {'__module__': 'gevent._hub_primitives', '__qualname__': 'WaitOperationsGreenlet', 'wait': <function at remote 0x7fffe790b280>, 'cancel_waits_close_and_then': <function at remote 0x7fffe790b330>, '_cancel_waits_then': <function at remote 0x7fffe790b3e0>, 'cancel_wait': <function at remote 0x7fffe790b490>, '_cancel_wait': <function at remote 0x7fffe790b540>}), kwds=0x0) at Objects/typeobject.c:1127
...

(gdb) py-bt
Traceback (most recent call first):
  <built-in method __build_class__ of module object at remote 0x7fffea697c50>
  File "/home/vstinner/bug/python-x2go-0.6.1.3/env/lib/python3.10/site-packages/gevent-21.1.2-py3.10-linux-x86_64.egg/gevent/_hub_primitives.py", line 44, in <module>
    class WaitOperationsGreenlet(SwitchOutGreenletWithLoop): # pylint:disable=undefined-variable
  <built-in method exec of module object at remote 0x7fffea697c50>
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "/home/vstinner/bug/python-x2go-0.6.1.3/env/lib/python3.10/site-packages/gevent-21.1.2-py3.10-linux-x86_64.egg/gevent/__init__.py", line 87, in <module>
    from gevent._hub_primitives import iwait_on_objects as iwait
(...)

Before the crash, I see warnings like that:

<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject

This warning comes from Cython __Pyx_ImportType() function.

Oh, at May 6, 2021, binary wheel files of greenlet for Python 3.10 have been published, even if Python 3.10 is still a beta version:
https://pypi.org/project/greenlet/#files


I simplified src/gevent/_greenlet_primitives.py to:
---
import greenlet._greenlet

class TrackedRawGreenlet(greenlet._greenlet.greenlet):
    ...

class SwitchOutGreenletWithLoop(TrackedRawGreenlet):
    ...
---

where greenlet._greenlet is a C extension.

Script to reproduce the crash, bug.py:
---
from gevent._gevent_c_greenlet_primitives import SwitchOutGreenletWithLoop

class WaitOperationsGreenlet(SwitchOutGreenletWithLoop):
    ...
---

Commands to reproduce the issue on Fedora 34:
---
mkdir bug; cd bug

# Development branch of Python 3.10 built in debug mode
~/python/3.10/python -m venv ../env
# Install Cython 0.29.23
env/bin/python -m pip install Cython 

wget https://files.pythonhosted.org/packages/47/6d/be10df2b141fcb1020c9605f7758881b5af706fb09a05b737e8eb7540387/greenlet-1.1.0.tar.gz
tar -xf greenlet-1.1.0.tar.gz && cd greenlet-1.1.0/
../env/bin/python setup.py install
cd ..

wget https://files.pythonhosted.org/packages/0b/50/1b1175ea3a269b5fa3f0f7fed11149888180695bef5fbf568adbb196efaf/gevent-21.1.2.tar.gz
tar -xf gevent-21.1.2.tar.gz && cd gevent-21.1.2/
../env/bin/python setup.py install
cd ..

./env/bin/python -c 'import gevent'
---

where ~/python/3.10/ is a checkout of the 3.10 branch of git:python/cpython.git and the ~/python/3.10/python was built with "./configure --with-pydebug CFLAGS=-O0 --with-system-expat --with-system-ffi && make".

Output:
---
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
<frozen importlib._bootstrap>:241: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 152 from C header, got 160 from PyObject
python: Objects/typeobject.c:2219: extra_ivars: Assertion `t_size >= b_size' failed.
Abandon (core dumped)
---

Oh, gevent 21.1.2 contains an outdated copy of greenlet.h in deps/greenlet/greenlet.h, the old copy is ABI incompatible with greenlet 1.1.0.

greenlet 1.1.0 contains a fix for Python 3.10:
https://greenlet.readthedocs.io/en/latest/changes.html#id1

gevent copy doesn't the greenlet fix for Python 3.10.


Example of __Pyx_ImportType() call in gevent-21.1.2/src/gevent/_hub_local.c, code run when the module is imported (Py_mod_exec slot of the module):
---
#include "greenlet/greenlet.h"
...
__pyx_ptype_6gevent_29_gevent_c_greenlet_primitives_greenlet = __Pyx_ImportType(__pyx_t_1, "greenlet", "greenlet", sizeof(PyGreenlet), __Pyx_ImportType_CheckSize_Warn);
---

In env/lib/python3.10/site-packages/greenlet-1.1.0-py3.10-linux-x86_64.egg/greenlet/greenlet.h, PyGreenlet structure is defined as:

typedef struct _greenlet {
    PyObject_HEAD
    char* stack_start;
    char* stack_stop;
    char* stack_copy;
    intptr_t stack_saved;
    struct _greenlet* stack_prev;
    struct _greenlet* parent;
    PyObject* run_info;
    struct _frame* top_frame;
    int recursion_depth;
    PyObject* weakreflist;
#if PY_VERSION_HEX >= 0x030700A3
    _PyErr_StackItem* exc_info;
    _PyErr_StackItem exc_state;
#else
    PyObject* exc_type;
    PyObject* exc_value;
    PyObject* exc_traceback;
#endif
    PyObject* dict;
#if PY_VERSION_HEX >= 0x030700A3
    PyObject* context;
#endif
#if PY_VERSION_HEX >= 0x30A00B1
    CFrame* cframe;
#endif
} PyGreenlet;

Cython compiles src/gevent/_hub_local.py to src/gevent/_hub_local.c.

GCC compiles src/gevent/_hub_local.c to src/gevent/_hub_local.o using -I/home/vstinner/bug/gevent-21.1.2/deps option.

Problem: gevent-21.1.2/deps/ contains a copy of greenlet/greenlet.h which is outdated:
---
typedef struct _greenlet {
    PyObject_HEAD
    char* stack_start;
    char* stack_stop;
    char* stack_copy;
    intptr_t stack_saved;
    struct _greenlet* stack_prev;
    struct _greenlet* parent;
    PyObject* run_info;
    struct _frame* top_frame;
    int recursion_depth;
    PyObject* weakreflist;
#if PY_VERSION_HEX >= 0x030700A3
    _PyErr_StackItem* exc_info;
    _PyErr_StackItem exc_state;
#else
    PyObject* exc_type;
    PyObject* exc_value;
    PyObject* exc_traceback;
#endif
    PyObject* dict;
#if PY_VERSION_HEX >= 0x030700A3
    PyObject* context;
#endif
} PyGreenlet;
---

It doesn't contain the last cframe member of Python 3.10 beta1 and newer.

> Oh, gevent 21.1.2 contains an outdated copy of greenlet.h in deps/greenlet/greenlet.h, the old copy is ABI incompatible with greenlet 1.1.0.

Maybe the gevent specfile should remove its vendored copy to get greenlet.h from a build dependency, to avoid such ABI issue.

> Maybe the gevent specfile should remove its vendored copy to get greenlet.h from a build dependency, to avoid such ABI issue. 

Oh, ignore my previous comments, the Fedora python-gevent package is not affected. It already removes deps/ directory in its specfile:
https://src.fedoraproject.org/rpms/python-gevent/blob/rawhide/f/python-gevent.spec
---
%prep	
%autosetup -p1 -n %{modname}-%{version}	
# Remove bundled libraries	
rm -r deps
---

I've deleted the build with Python 3.10.0b1 from https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-gevent/ and verified the only latest successful build was done with 3.10.0b2.

I also see a successful build of https://copr.fedorainfracloud.org/coprs/g/python/python3.10/package/python-x2go/

This seems like a simple matter of misscompilation with an older Python 3.10 pre-release. Stuff like that happens in our Copr and we have no easy way to solve this, except to keep rebuilding everything until it does. Sorry for the noise.