Bug 2118215

Summary:	Segmentation fault in _PyDictKeys_StringLookup() during sphinx-build
Product:	[Fedora] Fedora	Reporter:	Orion Poplawski <orion>
Component:	python3.11	Assignee:	Python Maintainers <python-maint>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	cstratak, mhroncok, negativo17, python-maint, python-sig, quantum.analyst, susi.lehtola, thrnciar, torsava, vstinner
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-09-06 09:52:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2121905

Description Orion Poplawski 2022-08-15 02:10:02 UTC

Description of problem:

I'm looking at updating python-pikepdf to 5.4.2 (https://src.fedoraproject.org/fork/orion/rpms/python-pikepdf/tree/5.4) but python is segfaulting during sphinx-build:

+ sphinx-build-3 . ../html
Running Sphinx v5.0.2
WARNING: Invalid configuration value found: 'language = None'. Update your configuration to a valid langauge code. Falling back to 'en' (English).
making output directory... done
WARNING: html_static_path entry '_static' does not exist
loading intersphinx inventory from https://docs.python.org/objects.inv...
intersphinx inventory has moved: https://docs.python.org/objects.inv -> https://docs.python.org/3/objects.inv
[autosummary] generating autosummary for: api/exceptions.rst, api/filters.rst, api/main.rst, api/models.rst, api/settings.rst, index.rst, installation.rst, references/arch.rst, references/contributing.rst, references/debugging.rst, ..., topics/nametrees.rst, topics/objects.rst, topics/outlines.rst, topics/overlays.rst, topics/page.rst, topics/pagelayout.rst, topics/pages.rst, topics/security.rst, topics/streams.rst, tutorial.rst
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 33 source files that are out of date
updating environment: [new config] 33 added, 0 changed, 0 removed
reading sources... [  3%] api/exceptions
reading sources... [  6%] api/filters
reading sources... [  9%] api/main
reading sources... [ 12%] api/models
reading sources... [ 15%] api/settings
reading sources... [ 18%] index
reading sources... [ 21%] installation
reading sources... [ 24%] references/arch
reading sources... [ 27%] references/contributing
reading sources... [ 30%] references/debugging
reading sources... [ 33%] references/resources
reading sources... [ 36%] releasenotes/index
reading sources... [ 39%] releasenotes/version0
reading sources... [ 42%] releasenotes/version1
reading sources... [ 45%] releasenotes/version2
reading sources... [ 48%] releasenotes/version3
reading sources... [ 51%] releasenotes/version4
reading sources... [ 54%] releasenotes/version5
reading sources... [ 57%] topics/attachments
reading sources... [ 60%] topics/content_streams
reading sources... [ 63%] topics/encoding
reading sources... [ 66%] topics/images
reading sources... [ 69%] topics/metadata
/var/tmp/rpm-tmp.pUjQ9C: line 71: 44020 Segmentation fault      (core dumped) sphinx-build-3 . ../html


Version-Release number of selected component (if applicable):
python3-3.11.0~rc1-1.fc38.x86_64

How reproducible:
Every time it seems

Additional info:
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00007ffff7bde6a6 in _PyDictKeys_StringLookup (dk=0x0, key=0x7ffff46dc170) at /usr/src/debug/python3.11-3.11.0~rc1-1.fc38.x86_64/Objects/dictobject.c:1011
Downloading 0.16 MB source file /usr/src/debug/python3.11-3.11.0~rc1-1.fc38.x86_64/Objects/dictobject.c
1011        if (!PyUnicode_CheckExact(key) || kind == DICT_KEYS_GENERAL) {                                                                                                                                                                 
(gdb) print key
$1 = (PyObject *) 0x7ffff46dc170
(gdb) print kind
$2 = <optimized out>
(gdb) print *key
$3 = {ob_refcnt = 8, ob_type = 0x7ffff7eed620 <PyUnicode_Type>}
(gdb) list
1006     */
1007    Py_ssize_t
1008    _PyDictKeys_StringLookup(PyDictKeysObject* dk, PyObject *key)
1009    {
1010        DictKeysKind kind = dk->dk_kind;
1011        if (!PyUnicode_CheckExact(key) || kind == DICT_KEYS_GENERAL) {
1012            return DKIX_ERROR;
1013        }
1014        Py_hash_t hash = unicode_get_hash(key);
1015        if (hash == -1) {

Comment 1 Elliott Sales de Andrade 2022-08-15 06:51:05 UTC

FYI, this is also broken with the existing 5.3.0, though you need to backport [1] to prevent dependency issues.

The previous working build was with python3-pybind11-2.9.2-2.fc37, though koschei did not track it due to the pin fixed by [1]. But locally, downgrading to that version works, so this appears to be a pybind11 issue.

[1] https://github.com/pikepdf/pikepdf/commit/cc977d343ce044fe021c545c8d8c6b159d477f6b

Comment 2 Victor Stinner 2022-08-17 15:49:51 UTC

I reproduce the crash on Fedora 36 with these commands:
---
sudo dnf install qpdf-devel

python3.11 -m venv env
cd env
source ./bin/activate

python -m pip install IPython
python -m pip install sphinx sphinx_issues sphinx_design sphinx_rtd_theme

git clone https://github.com/pikepdf/pikepdf
cd pikepdf/
python -m pip install .

cd docs/
~/env/bin/sphinx-build . ../html
---

Output:
---
(env) vstinner@mona$ ~/env/bin/python -X dev ~/env/bin/sphinx-build . ../html
(...)
Fatal Python error: Segmentation faultetadata                                                                                                                                          

Current thread 0x00007fcbb8ca0740 (most recent call first):
  File "/home/vstinner/env/lib64/python3.11/site-packages/pikepdf/_methods.py", line 798 in open
  File "<ipython-input-5-851f84133ed8>", line 1 in <cell line: 0>
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/core/interactiveshell.py", line 3398 in run_code
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/core/interactiveshell.py", line 3338 in run_ast_nodes
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/core/interactiveshell.py", line 3135 in run_cell_async
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/core/async_helpers.py", line 129 in _pseudo_sync_runner
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/core/interactiveshell.py", line 2936 in _run_cell
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/core/interactiveshell.py", line 2881 in run_cell
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/sphinxext/ipython_directive.py", line 411 in process_input_lines
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/sphinxext/ipython_directive.py", line 497 in process_input
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/sphinxext/ipython_directive.py", line 728 in process_block
  File "/home/vstinner/env/lib64/python3.11/site-packages/IPython/sphinxext/ipython_directive.py", line 1025 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 2146 in run_directive
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 2096 in directive
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 2354 in explicit_construct
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 2342 in explicit_markup
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/statemachine.py", line 451 in check_line
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/statemachine.py", line 239 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 196 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 281 in nested_parse
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 393 in new_subsection
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 327 in section
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 2769 in underline
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/statemachine.py", line 451 in check_line
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/statemachine.py", line 239 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 196 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 281 in nested_parse
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 393 in new_subsection
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 327 in section
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 2769 in underline
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/statemachine.py", line 451 in check_line
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/statemachine.py", line 239 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/parsers/rst/states.py", line 170 in run
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/parsers.py", line 78 in parse
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/readers/__init__.py", line 78 in parse
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/io.py", line 104 in read
  File "/home/vstinner/env/lib64/python3.11/site-packages/docutils/core.py", line 217 in publish
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/builders/__init__.py", line 498 in read_doc
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/builders/__init__.py", line 445 in _read_serial
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/builders/__init__.py", line 424 in read
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/builders/__init__.py", line 317 in build
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/builders/__init__.py", line 301 in build_update
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/application.py", line 349 in build
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/cmd/build.py", line 277 in build_main
  File "/home/vstinner/env/lib64/python3.11/site-packages/sphinx/cmd/build.py", line 291 in main
  File "/home/vstinner/env/bin/sphinx-build", line 8 in <module>

Extension modules: markupsafe._speedups, PIL._imaging, PIL._imagingcms, lxml._elementpath, lxml.etree (total: 5)
Erreur de segmentation (core dumped)
---

Comment 3 Victor Stinner 2022-08-17 16:07:17 UTC

I reported the issue to Python upstream: https://github.com/python/cpython/issues/96046

Comment 4 Victor Stinner 2022-09-06 09:52:36 UTC

Python 3.11 is now fixed: pybind11 no longer crash on Python 3.11. See latest comments of the upstream issue: https://github.com/python/cpython/issues/96046

Comment 5 Victor Stinner 2022-09-06 09:53:38 UTC

Upstream fix (3.11 branch): https://github.com/python/cpython/commit/3ea0beb3599f734bb9387a526dccd5768ad6b1a9