Bug 2186412

Summary: sudo fails to build with Python 3.12: FAILED: testcase check_example_group_plugin_is_able_to_debug()
Product: [Fedora] Fedora Reporter: Tomáš Hrnčiar <thrnciar>
Component: sudoAssignee: Radovan Sroka <rsroka>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rawhideCC: alakatos, kzak, mattdm, mhroncok, pviktori, rsroka, thrnciar, tosykora, yselkowi, zfridric
Target Milestone: ---Flags: mhroncok: needinfo? (pviktori)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: https://bugzilla.sudo.ws/show_bug.cgi?id=1054
Whiteboard:
Fixed In Version: sudo-1.9.14-1.p3.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-26 10:24:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2135404, 2168842    

Description Tomáš Hrnčiar 2023-04-13 08:16:53 UTC
sudo fails to build with Python 3.12.0a7.

Expectation failed at ./regress/check_python_examples.c:512:
  actual is <<0>>: verify_log_lines(expected_path("check_example_group_plugin_is_able_to_debug.log"))
  not expected to be <<0>>: 0
FAILED: testcase check_example_group_plugin_is_able_to_debug() at ./regress/check_python_examples.c:1543

https://docs.python.org/3.12/whatsnew/3.12.html

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.12/fedora-rawhide-x86_64/05772134-sudo/

For all our attempts to build sudo with Python 3.12, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/package/sudo/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.12:
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/

Let us know here if you have any questions.

Python 3.12 is planned to be included in Fedora 39. To make that update smoother, we're building Fedora packages with all pre-releases of Python 3.12.
A build failure prevents us from testing all dependent packages (transitive [Build]Requires), so if this package is required a lot, it's important for us to get it fixed soon.
We'd appreciate help from the people who know this package best, but if you don't want to work on this now, let us know so we can try to work around it on our side.

Comment 1 Miro Hrončok 2023-07-10 21:45:17 UTC
Current failure:


make[1]: Entering directory '/builddir/build/BUILD/sudo-1.9.13p2/plugins/python'
gcc -c -I../../include -I../.. -I../.. -DPLUGIN_DIR=\"/usr/libexec/sudo\" -DSRC_DIR=\"/builddir/build/BUILD/sudo-1.9.13p2/plugins/python\" -DZLIB_CONST -D_FORTIFY_SOURCE=2 -I/usr/include/python3.12  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpie -fvisibility=hidden  -fPIE -fstack-protector-strong -fstack-clash-protection -fcf-protection ./regress/check_python_examples.c
gcc -c -I../../include -I../.. -I../.. -DPLUGIN_DIR=\"/usr/libexec/sudo\" -DSRC_DIR=\"/builddir/build/BUILD/sudo-1.9.13p2/plugins/python\" -DZLIB_CONST -D_FORTIFY_SOURCE=2 -I/usr/include/python3.12  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpie -fvisibility=hidden  -fPIE -fstack-protector-strong -fstack-clash-protection -fcf-protection ./regress/iohelpers.c
gcc -c -I../../include -I../.. -I../.. -DPLUGIN_DIR=\"/usr/libexec/sudo\" -DSRC_DIR=\"/builddir/build/BUILD/sudo-1.9.13p2/plugins/python\" -DZLIB_CONST -D_FORTIFY_SOURCE=2 -I/usr/include/python3.12  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpie -fvisibility=hidden  -fPIE -fstack-protector-strong -fstack-clash-protection -fcf-protection ./regress/testhelpers.c
gcc -c -I../../include -I../.. -I../.. -DPLUGIN_DIR=\"/usr/libexec/sudo\" -DSRC_DIR=\"/builddir/build/BUILD/sudo-1.9.13p2/plugins/python\" -DZLIB_CONST -D_FORTIFY_SOURCE=2 -I/usr/include/python3.12  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpie -fvisibility=hidden  -fPIE -fstack-protector-strong -fstack-clash-protection -fcf-protection ./pyhelpers.c
gcc -c -I../../include -I../.. -I../.. -DPLUGIN_DIR=\"/usr/libexec/sudo\" -DSRC_DIR=\"/builddir/build/BUILD/sudo-1.9.13p2/plugins/python\" -DZLIB_CONST -D_FORTIFY_SOURCE=2 -I/usr/include/python3.12  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpie -fvisibility=hidden  -fPIE -fstack-protector-strong -fstack-clash-protection -fcf-protection ./sudo_python_debug.c
/bin/sh ../../libtool --tag=disable-static --mode=link gcc -o check_python_examples check_python_examples.o iohelpers.o testhelpers.o pyhelpers.o sudo_python_debug.o -pie -Wl,-z,relro -Wl,-z,now -Wl,--no-undefined -Wl,--enable-new-dtags -L/usr/lib64 -lpython3.12 -ldl -lm  -Wl,--allow-multiple-definition -Wc,-fPIE -pie -Wc,-fstack-protector-strong -Wc,-fstack-clash-protection -Wc,-fcf-protection -Wl,-z,relro -Wl,-z,now -Wl,-z,noexecstack ../../lib/util/libsudo_util.la
libtool: link: gcc -o .libs/check_python_examples check_python_examples.o iohelpers.o testhelpers.o pyhelpers.o sudo_python_debug.o -pie -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,--no-undefined -Wl,--enable-new-dtags -Wl,--allow-multiple-definition -fPIE -pie -fstack-protector-strong -fstack-clash-protection -fcf-protection -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-z -Wl,noexecstack  -L/usr/lib64 -lpython3.12 -ldl -lm ../../lib/util/.libs/libsudo_util.so -lcrypto -lpthread -Wl,-rpath -Wl,/usr/libexec/sudo
make[1]: *** [Makefile:231: check] Error 139
make[1]: Leaving directory '/builddir/build/BUILD/sudo-1.9.13p2/plugins/python'
make: *** [Makefile:114: check] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.B4lDuQ (%check)

Comment 2 Yaakov Selkowitz 2023-07-12 14:38:58 UTC
Backtrace:

#0  _PyObject_GC_UNTRACK (op=0x7ffff52f9d00)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Include/internal/pycore_object.h:245
#1  descr_dealloc (descr=0x7ffff52f9d00)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Objects/descrobject.c:39
#2  0x00007ffff7a834e5 in _PyEval_EvalFrameDefault (tstate=<optimized out>, 
    frame=<optimized out>, throwflag=<optimized out>)
    at Python/bytecodes.c:3029
#3  0x00007ffff7b91a57 in _PyObject_VectorcallTstate (kwnames=0x0, 
    nargsf=9223372036854775809, args=0x7fffffffdaf8, callable=0x7ffff6e7e700, 
    tstate=0x7ffff55f08d0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Include/internal/pycore_call.h:92
#4  PyObject_CallOneArg (func=0x7ffff6e7e700, arg=<optimized out>)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Objects/call.c:401
#5  0x00007ffff7b7846e in handle_weakrefs (old=0x7ffff55d3758, 
    unreachable=0x7fffffffdba0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Modules/gcmodule.c:899
#6  gc_collect_main (tstate=tstate@entry=0x7ffff55f08d0, 
    generation=generation@entry=2, n_collected=n_collected@entry=0x0, 
    n_uncollectable=n_uncollectable@entry=0x0, nofail=nofail@entry=1)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Modules/gcmodule.c:1284--Type <RET> for more, q to quit, c to continue without paging--

#7  0x00007ffff7c33291 in _PyGC_CollectNoFail (
    tstate=tstate@entry=0x7ffff55f08d0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Modules/gcmodule.c:2135
#8  0x00007ffff7c35ab1 in interpreter_clear (interp=0x7ffff5593010, 
    tstate=tstate@entry=0x7ffff55f08d0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Python/pystate.c:877
#9  0x00007ffff7bc4d30 in _PyInterpreterState_Clear (tstate=0x7ffff55f08d0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Python/pystate.c:935
#10 finalize_interp_clear (tstate=tstate@entry=0x7ffff55f08d0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Python/pylifecycle.c:1717
#11 0x00007ffff7c40188 in Py_EndInterpreter (tstate=0x7ffff55f08d0)
    at /usr/src/debug/python3.12-3.12.0~b3-2.fc39.x86_64/Python/pylifecycle.c:2174
#12 0x00007ffff71dad60 in python_plugin_unlink ()
    at ./python_plugin_common.c:738
#13 0x00007ffff7fcb0f2 in _dl_call_fini (
    closure_map=closure_map@entry=0x5555555d0390) at dl-call_fini.c:43
#14 0x00007ffff7fcb5c2 in __GI__dl_catch_exception (
    exception=exception@entry=0x0, operate=0x7ffff7fcb090 <_dl_call_fini>, 
    args=args@entry=0x5555555d0390) at dl-catch.c:211
--Type <RET> for more, q to quit, c to continue without paging--
#15 0x00007ffff7fcbd69 in _dl_close_worker (map=map@entry=0x5555555d0390, 
    force=force@entry=false) at dl-close.c:253
#16 0x00007ffff7fcc5eb in _dl_close (_map=0x5555555d0390) at dl-close.c:778
#17 0x00007ffff7fcb523 in __GI__dl_catch_exception (
    exception=exception@entry=0x7fffffffe150, 
    operate=0x7ffff7fcc5b0 <_dl_close>, args=0x5555555d0390) at dl-catch.c:237
#18 0x00007ffff7fcb679 in _dl_catch_error (objname=0x7fffffffe1b8, 
    errstring=0x7fffffffe1c0, mallocedp=0x7fffffffe1b7, 
    operate=<optimized out>, args=<optimized out>) at dl-catch.c:256
#19 0x00007ffff72963e3 in _dlerror_run (operate=<optimized out>, 
    args=<optimized out>) at dlerror.c:138
#20 0x00007ffff7296116 in __dlclose (handle=<optimized out>) at dlclose.c:31
#21 0x00007ffff787ddbf in sudo_dso_unload_v1 (handle=<optimized out>)
    at ./sudo_dso.c:315
#22 0x000055555555b75c in _unlink_symbols ()
    at ./regress/check_python_examples.c:1501
#23 check_plugin_unload () at ./regress/check_python_examples.c:525
#24 0x000055555555918e in main (argc=<optimized out>, argv=<optimized out>)
    at ./regress/check_python_examples.c:1539

Comment 3 Yaakov Selkowitz 2023-07-18 21:37:13 UTC
Still crashes with 3.12.0b4; updating sudo to latest upstream 1.9.14p2 doesn't help.

Reported upstream: https://bugzilla.sudo.ws/show_bug.cgi?id=1054

Comment 4 Yaakov Selkowitz 2023-07-20 04:20:23 UTC
sudo upstream believes this may be a bug in Python itself:

https://bugzilla.sudo.ws/show_bug.cgi?id=1054#c1

Please advise.

Comment 5 Yaakov Selkowitz 2023-07-20 15:10:44 UTC
This is a blocker to the F39 mass rebuild and CentOS Stream 10 initial import.

Comment 6 Miro Hrončok 2023-07-20 15:25:46 UTC
Almost the entire Python Maint team is at EuroPython this week. Petr, could you please have a look?

(A package that fails to build is usually not a blocker for the Fedora mass rebuild -- even when it's important.)

Comment 7 Yaakov Selkowitz 2023-07-23 20:36:33 UTC
Upstream has applied a series of patches which allow the tests to pass, but still suspect an underlying issue somewhere.

Filed downstream PR: https://src.fedoraproject.org/rpms/sudo/pull-request/26

Comment 8 Miro Hrončok 2023-07-25 09:39:30 UTC
We are back from EuroPython. Thanks for working with upstream sudo to make it build!

https://src.fedoraproject.org/rpms/sudo/pull-request/26

I'll leave the needinfo for Petr so we don't forget to investigate the Python issue but we won't treat it urgently.

Comment 9 Fedora Update System 2023-07-26 08:59:44 UTC
FEDORA-2023-4a0fbe40e1 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-4a0fbe40e1

Comment 10 Fedora Update System 2023-07-26 10:24:04 UTC
FEDORA-2023-4a0fbe40e1 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.