I'm seeing a GDB crash on Fedora 37, Fedora 38, and rawhide (f39). I think that the bug is actually in one of the openssl packages. When the default /etc/ssl/openssl.cnf file is used, the bug does not occur. That said, I'm going to start with the component set to "gdb" instead of "openssl". The file /etc/ssl/openssl.cnf contains the following lines: [provider_sect] default = default_sect ##legacy = legacy_sect ## [default_sect] activate = 1 ##[legacy_sect] ##activate = 1 The bug will occur when the '##' characters are removed so that the lines in question look like this: [provider_sect] default = default_sect legacy = legacy_sect [default_sect] activate = 1 [legacy_sect] activate = 1 With that change in place, do the following to reproduce the bug: [kev@f38-2 e]$ rm -rf $HOME/.cache/debuginfod_client [kev@f38-2 e]$ gdb -q /bin/ls Reading symbols from /bin/ls... This GDB supports auto-downloading debuginfo from the following URLs: <https://debuginfod.fedoraproject.org/> Enable debuginfod for this session? (y or [n]) y Debuginfod has been enabled. To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit. Downloading separate debug info for /usr/bin/ls Reading symbols from /home/kev/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo... Downloading separate debug info for /home/kev/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo (gdb) start Downloading source file /usr/src/debug/coreutils-9.1-11.fc38.x86_64/separate/../src/ls.c Temporary breakpoint 1 at 0x4e50: file ../src/ls.c, line 1646. Starting program: /usr/bin/ls Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000 Downloading separate debug info for /lib64/libselinux.so.1 Downloading separate debug info for /home/kev/.cache/debuginfod_client/7adfc95d3eb203b795825f82ab41639cb0bb9c4c/debuginfo Downloading separate debug info for /lib64/libcap.so.2 Downloading separate debug info for /home/kev/.cache/debuginfod_client/2fe8ebb5811811046e04d8f4c000696cb9b24e24/debuginfo [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Downloading separate debug info for /lib64/libpcre2-8.so.0 Downloading separate debug info for /home/kev/.cache/debuginfod_client/1ef098ba40305d86ec0cdae1e6aac188b1656550/debuginfo Temporary breakpoint 1, main (argc=1, argv=0x7fffffffda68) at ../src/ls.c:1646 1646 { (gdb) quit A debugging session is active. Inferior 1 [process 59528] will be killed. Quit anyway? (y or n) y Fatal signal: Segmentation fault ----- Backtrace ----- 0x55a251ed9810 gdb_internal_backtrace_1 ../../gdb/bt-utils.c:122 0x55a251ed9810 _Z22gdb_internal_backtracev ../../gdb/bt-utils.c:168 0x55a2520140dd handle_fatal_signal ../../gdb/event-top.c:962 0x55a2520142a9 handle_sigsegv ../../gdb/event-top.c:1035 0x7f542b23bb6f ??? /usr/src/debug/glibc-2.37-4.fc38.x86_64/signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 0x7f542b2902b5 ??? /usr/src/debug/glibc-2.37-4.fc38.x86_64/nptl/pthread_rwlock_common.c:298 0x7f54283c0f8c ??? 0x7f54283b901e ??? 0x7f54283c659c ??? 0x7f54283b9f46 ??? 0x7f54180e7bed ??? 0x7f54283c69fa ??? 0x7f54283904bb ??? 0x7f5428b96abc ??? 0x7f542af5c306 ??? ../../lib/vtls/openssl.c:1895 0x7f542af661ce ??? ../../lib/vtls/vtls.c:1423 0x7f542af02203 ??? ../../lib/cfilters.c:165 0x7f542af5ea92 ??? ../../lib/cfilters.c:186 0x7f542af0369b ??? ../../lib/conncache.c:532 0x7f542af3c9b2 ??? ../../lib/multi.c:2801 0x7f542b73c54d ??? /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1892 0x55a251f7d9c4 _ZN25debuginfod_client_deleterclEP17debuginfod_client ../../gdb/debuginfod-support.c:105 0x55a251f7d9c4 _ZNSt10unique_ptrI17debuginfod_client25debuginfod_client_deleterED2Ev /usr/include/c++/13/bits/unique_ptr.h:404 0x7f542b23e1a5 ??? /usr/src/debug/glibc-2.37-4.fc38.x86_64/stdlib/exit.c:111 0x7f542b23e2ed __GI_exit /usr/src/debug/glibc-2.37-4.fc38.x86_64/stdlib/exit.c:141 0x55a252313099 _Z10quit_forcePii ../../gdb/top.c:1894 0x55a251f0c14b _Z12quit_commandPKci ../../gdb/cli/cli-cmds.c:506 0x55a251f14064 _Z8cmd_funcP16cmd_list_elementPKci ../../gdb/cli/cli-decode.c:2543 0x55a2523142a4 _Z15execute_commandPKci ../../gdb/top.c:688 0x55a2520148de _Z15command_handlerPKc ../../gdb/event-top.c:619 0x55a252016398 _Z20command_line_handlerOSt10unique_ptrIcN3gdb13xfree_deleterIcEEE ../../gdb/event-top.c:855 0x55a252014f10 gdb_rl_callback_handler ../../gdb/event-top.c:247 0x7f542bf0a0cc ??? 0x55a252015075 gdb_rl_callback_read_char_wrapper_noexcept ../../gdb/event-top.c:189 0x55a252015227 gdb_rl_callback_read_char_wrapper ../../gdb/event-top.c:222 0x55a252013f8f stdin_event_handler ../../gdb/event-top.c:544 0x55a25257397d gdb_wait_for_event ../../gdbsupport/event-loop.cc:716 0x55a252574531 _Z16gdb_do_one_eventi ../../gdbsupport/event-loop.cc:264 0x55a2520fb481 start_event_loop ../../gdb/main.c:411 0x55a2520fb481 captured_command_loop ../../gdb/main.c:471 0x55a2520fd5f4 captured_main ../../gdb/main.c:1330 0x55a2520fd5f4 _Z8gdb_mainP18captured_main_args ../../gdb/main.c:1345 0x55a251dea9ee main ../../gdb/gdb.c:40 --------------------- A fatal error internal to GDB has been detected, further debugging is not possible. GDB will now terminate. This is a bug, please report it. For instructions, see: <https://www.gnu.org/software/gdb/bugs/>. Segmentation fault (core dumped) Reproducible: Always The backtrace for the core dump (on Fedora 38) looks like this: (gdb) bt #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44 #1 0x00007f542b28c8b3 in __pthread_kill_internal (signo=11, threadid=<optimized out>) at pthread_kill.c:78 #2 0x00007f542b23babe in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26 #3 0x000055a25201408a in handle_fatal_signal (sig=sig@entry=11) at ../../gdb/event-top.c:985 #4 0x000055a2520142aa in handle_sigsegv (sig=11) at ../../gdb/event-top.c:1035 #5 <signal handler called> #6 __pthread_rwlock_rdlock_full64 (abstime=0x0, clockid=0, rwlock=0x0) at /usr/src/debug/glibc-2.37-4.fc38.x86_64/nptl/pthread_rwlock_common.c:298 #7 ___pthread_rwlock_rdlock (rwlock=0x0) at pthread_rwlock_rdlock.c:26 #8 0x00007f54283c0f8d in CRYPTO_THREAD_read_lock (lock=<optimized out>) at crypto/threads_pthread.c:97 #9 0x00007f54283b901f in ossl_lib_ctx_get_data (ctx=<optimized out>, index=1, meth=0x7f54285ddaa0 <provider_store_method.lto_priv>) at crypto/context.c:413 #10 0x00007f54283c659d in get_provider_store (libctx=<optimized out>) at crypto/provider_core.c:339 #11 ossl_provider_deregister_child_cb (handle=0x55a2542fb0a0) at crypto/provider_core.c:1806 #12 0x00007f54283b9f47 in ossl_provider_deinit_child (ctx=0x55a2542f91b0) at crypto/provider_child.c:281 #13 OSSL_LIB_CTX_free (ctx=0x55a2542f91b0) at crypto/context.c:284 #14 OSSL_LIB_CTX_free (ctx=0x55a2542f91b0) at crypto/context.c:277 #15 0x00007f54180e7bee in legacy_teardown (provctx=0x55a254301fe0) at providers/legacyprov.c:168 #16 0x00007f54283c69fb in ossl_provider_teardown (prov=0x55a2542fb0a0) at crypto/provider_core.c:1558 #17 ossl_provider_free (prov=0x55a2542fb0a0) at crypto/provider_core.c:696 #18 0x00007f54283904bc in ossl_provider_free (prov=<optimized out>) at crypto/provider_core.c:681 #19 evp_cipher_free_int (cipher=0x55a25433dff0) at crypto/evp/evp_enc.c:1651 #20 EVP_CIPHER_free (cipher=0x55a25433dff0) at crypto/evp/evp_enc.c:1666 #21 0x00007f5428b96abd in ssl_evp_cipher_free (cipher=0x55a25433dff0) at ssl/ssl_lib.c:5967 #22 ssl_evp_cipher_free (cipher=0x55a25433dff0) at ssl/ssl_lib.c:5957 #23 SSL_CTX_free (a=0x55a254347620) at ssl/ssl_lib.c:3492 #24 SSL_CTX_free (a=0x55a254347620) at ssl/ssl_lib.c:3429 #25 0x00007f542af5c307 in ossl_close (cf=<optimized out>, data=<optimized out>) at ../../lib/vtls/openssl.c:1895 #26 0x00007f542af661cf in cf_close (data=<optimized out>, cf=0x55a254301050) at ../../lib/vtls/vtls.c:1423 #27 ssl_cf_destroy (cf=0x55a254301050, data=<optimized out>) at ../../lib/vtls/vtls.c:1483 #28 0x00007f542af02204 in Curl_conn_cf_discard_chain (pcf=<optimized out>, data=0x55a254304ab0) at ../../lib/cfilters.c:165 #29 0x00007f542af5ea93 in Curl_conn_close (index=0, data=0x55a254304ab0) at ../../lib/cfilters.c:186 #30 conn_shutdown (data=0x55a254304ab0) at ../../lib/url.c:692 #31 Curl_disconnect (data=0x55a254304ab0, conn=0x55a2542fcd80, dead_connection=<optimized out>) at ../../lib/url.c:803 #32 0x00007f542af0369c in Curl_conncache_close_all_connections ( connc=0x55a254300a68) at ../../lib/conncache.c:532 #33 0x00007f542af3c9b3 in curl_multi_cleanup (multi=0x55a254300960) at ../../lib/multi.c:2801 #34 curl_multi_cleanup (multi=0x55a254300960) at ../../lib/multi.c:2763 #35 0x00007f542b73c54e in debuginfod_end (client=0x55a254300290) at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1892 #36 debuginfod_end (client=0x55a254300290) at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1887 #37 0x000055a251f7d9c5 in debuginfod_client_deleter::operator() ( c=<optimized out>, this=<optimized out>) at ../../gdb/debuginfod-support.c:105 #38 std::unique_ptr<debuginfod_client, debuginfod_client_deleter>::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/unique_ptr.h:404 #39 0x00007f542b23e1a6 in __run_exit_handlers (status=0, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:111 #40 0x00007f542b23e2ee in __GI_exit (status=<optimized out>) at exit.c:141 #41 0x000055a25231309a in quit_force (exit_arg=<optimized out>, from_tty=from_tty@entry=1) at ../../gdb/top.c:1894 #42 0x000055a251f0c14c in quit_command (args=0x0, from_tty=1) at ../../gdb/cli/cli-cmds.c:506 #43 0x000055a251f14065 in cmd_func (cmd=<optimized out>, args=<optimized out>, from_tty=<optimized out>) at ../../gdb/cli/cli-decode.c:2543 #44 0x000055a2523142a5 in execute_command (p=<optimized out>, p@entry=0x55a253f85c70 "quit", from_tty=<optimized out>) at ../../gdb/top.c:688 #45 0x000055a2520148df in command_handler (command=0x55a253f85c70 "quit") at ../../gdb/event-top.c:619 #46 0x000055a252016399 in command_line_handler (rl=...) at ../../gdb/event-top.c:855 #47 0x000055a252014f11 in gdb_rl_callback_handler (rl=0x55a254e3e1c0 "quit") at ../../gdb/event-top.c:247 #48 0x00007f542bf0a0cd in rl_callback_read_char () at ../callback.c:299 #49 0x000055a252015076 in gdb_rl_callback_read_char_wrapper_noexcept () at ../../gdb/event-top.c:189 #50 0x000055a252015228 in gdb_rl_callback_read_char_wrapper ( client_data=<optimized out>) at ../../gdb/event-top.c:221 #51 0x000055a252013f90 in stdin_event_handler (error=<optimized out>, client_data=0x55a253f91290) at ../../gdb/event-top.c:544 #52 0x000055a25257397e in gdb_wait_for_event (block=block@entry=1) at ../../gdbsupport/event-loop.cc:716 #53 0x000055a252574532 in gdb_do_one_event (mstimeout=mstimeout@entry=-1) at ../../gdbsupport/event-loop.cc:264 #54 0x000055a2520fb482 in start_event_loop () at ../../gdb/main.c:411 #55 captured_command_loop () at ../../gdb/main.c:471 #56 0x000055a2520fd5f5 in captured_main (data=0x7ffe0d8fab60) at ../../gdb/main.c:1330 #57 gdb_main (args=args@entry=0x7ffe0d8fab90) at ../../gdb/main.c:1345 #58 0x000055a251dea9ef in main (argc=<optimized out>, argv=<optimized out>) at ../../gdb/gdb.c:40
This is likely an openssl issue so I'm changing the component. I'm not seeing anything suspect on gdb's side. According to valgrind the cause of the crash is an invalid read: ==89655== Invalid read of size 4 ==89655== at 0x545D7FC: UnknownInlinedFun (pthread_rwlock_common.c:298) ==89655== by 0x545D7FC: pthread_rwlock_rdlock@@GLIBC_2.34 (pthread_rwlock_rdlock.c:26) ==89655== by 0x59D87AC: CRYPTO_THREAD_read_lock (threads_pthread.c:97) ==89655== by 0x59D0D95: ossl_lib_ctx_get_data (context.c:377) ==89655== by 0x59DD989: UnknownInlinedFun (provider_core.c:339) ==89655== by 0x59DD989: ossl_provider_deregister_child_cb (provider_core.c:1806) ==89655== by 0x59D1B5E: UnknownInlinedFun (provider_child.c:281) ==89655== by 0x59D1B5E: UnknownInlinedFun (context.c:248) ==89655== by 0x59D1B5E: OSSL_LIB_CTX_free (context.c:241) ==89655== by 0x1BBAEBC5: legacy_teardown (legacyprov.c:168) ==89655== by 0x59DDDBA: UnknownInlinedFun (provider_core.c:1558) ==89655== by 0x59DDDBA: ossl_provider_free.part.0 (provider_core.c:696) ==89655== by 0x59A925B: UnknownInlinedFun (provider_core.c:681) ==89655== by 0x59A925B: UnknownInlinedFun (evp_enc.c:1651) ==89655== by 0x59A925B: EVP_CIPHER_free (evp_enc.c:1666) ==89655== by 0x57B32FC: UnknownInlinedFun (ssl_lib.c:5967) ==89655== by 0x57B32FC: UnknownInlinedFun (ssl_lib.c:5957) ==89655== by 0x57B32FC: UnknownInlinedFun (ssl_lib.c:3492) ==89655== by 0x57B32FC: SSL_CTX_free (ssl_lib.c:3429) ==89655== by 0x5626F8D: ossl_closeone.isra.0 (openssl.c:1462) ==89655== by 0x5616135: ossl_close.lto_priv.0 (openssl.c:1473) ==89655== by 0x560CAC4: UnknownInlinedFun (vtls.c:696) ==89655== by 0x560CAC4: UnknownInlinedFun (url.c:758) ==89655== by 0x560CAC4: Curl_disconnect (url.c:879) ==89655== Address 0x18 is not stack'd, malloc'd or (recently) free'd
I'd say the problem is in Curl. There were similar bug reports - https://github.com/nmap/nmap/issues/2482 fixed by https://github.com/nmap/nmap/commit/5f88cbac30c438587de6d01c13327f9b7316c94f
Any idea how to reproduce it off the gdb process? I tried the following command and it did not crash: $ rm -rf ~/.cache/debuginfod_client && valgrind -q debuginfod-find -v executable /bin/ls Progress 1 / 0 Progress 7 / 0 Progress 8 / 0 Progress 9 / 0 Progress 10 / 0 Progress 11 / 0 Progress 12 / 0 Progress 40490 / 0 Headers: x-debuginfod-size: 142104 x-debuginfod-archive: /mnt/fedora_koji_prod/koji/packages/coreutils/9.1/11.fc38/x86_64/coreutils-9.1-11.fc38.x86_64.rpm x-debuginfod-file: /usr/bin/ls Downloaded from https://debuginfod.fedoraproject.org/buildid/813d7726e4ddb91b7f102d567852077c51f793c8/executable /home/kdudka/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/executable ... while the following command does: $ rm -rf ~/.cache/debuginfod_client && gdb -q /bin/ls <<< '' Reading symbols from /bin/ls... Downloading separate debug info for /usr/bin/ls... Reading symbols from /home/kdudka/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo... Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/813d7726e4ddb91b7f102d567852077c51f793c8/debuginfo... (gdb) (gdb) Fatal signal: Segmentation fault [...]
(In reply to Kamil Dudka from comment #3) > Any idea how to reproduce it off the gdb process? I have not been able to reproduce it outside of gdb. At first I thought it may have to do with gdb performing multiple queries with a single CURLM multi handle. But even when I ensure that gdb only downloads the ls debuginfo and nothing else (not even ls' .dwz), I still get a crash. The crash does not occur when gdb downloads from a local debuginfod server using a localhost URL. Strangely the local debuginfod server itself has no problem actually downloading the debuginfo on behalf of gdb. In this case there is no crash and valgrind does not detect any memory errors in either gdb or the debuginfod server. In any case no issues have been detected when using the default openssl.cnf.
I tried to run gdb through gdb with preloaded libcurl.so from the upstream git HEAD (f7170a8f2ed4dc5a4cfb3ef3c002d218c4bcecad) and the backtrace looks like this: $ rm -rf $HOME/.cache/debuginfod_client && LD_LIBRARY_PATH=/home/kdudka/curl/lib/.libs gdb -q -ex run --args gdb -q /bin/ls Reading symbols from gdb... Reading symbols from /usr/lib/debug/usr/libexec/gdb-13.1-4.fc38.x86_64.debug... Starting program: /usr/bin/gdb -q /bin/ls Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000 Downloading separate debug info for /lib64/liblzma.so.5 Downloading separate debug info for /lib64/libbabeltrace.so.1 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/c7cd104410d857eb743a49d645b8b703e66b0995/debuginfo Downloading separate debug info for /lib64/libbabeltrace-ctf.so.1 Downloading separate debug info for /lib64/libipt.so.2 Downloading separate debug info for /lib64/libmpfr.so.6 Downloading separate debug info for /lib64/libgmp.so.10 Downloading separate debug info for /lib64/libxxhash.so.0 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/3d77f993c64f6d4f487ca5bf7c8483149e6bd28e/debuginfo Downloading separate debug info for /lib64/libreadline.so.8 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/da79378d50e73c9cbf3405fdc55705835e281313/debuginfo Downloading separate debug info for /lib64/libz.so.1 Downloading separate debug info for /lib64/libzstd.so.1 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/95fc1c046547d542d2913ebc72290732112d8b9b/debuginfo Downloading separate debug info for /lib64/libncursesw.so.6 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/ce6b29b0c0bd3bbd68d3a87c7e51fd6a67d13f5e/debuginfo Downloading separate debug info for /lib64/libtinfo.so.6 Downloading separate debug info for /lib64/libpython3.11.so.1.0 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/486392b9c965505e765de7bc81c249eaa9ccc414/debuginfo Downloading separate debug info for /lib64/libexpat.so.1 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/1f06ca43854228d1ad3508eb0a8c4fb59f1f5c8e/debuginfo Downloading separate debug info for /lib64/libsource-highlight.so.4 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/22a663096876815f8215534dbc2f6f218f5f3388/debuginfo Downloading separate debug info for /lib64/libstdc++.so.6 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/4fbdf6ca060f2fe96c2321d9d59d775fa4b2ba3a/debuginfo Downloading separate debug info for /lib64/libm.so.6 Downloading separate debug info for /lib64/libgcc_s.so.1 Downloading separate debug info for /lib64/libc.so.6 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/245240a31888ad5c11bbc55b18e02d87388f59a9/debuginfo [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Downloading separate debug info for /lib64/libpopt.so.0 Downloading separate debug info for /lib64/libuuid.so.1 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/7f68b06dd9ec5a0f937cb1e19b9bc66f14997db1/debuginfo Downloading separate debug info for /lib64/libgmodule-2.0.so.0 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/12c007ff846734d4be86d9c1892f2ab3431ad824/debuginfo Downloading separate debug info for /lib64/libglib-2.0.so.0 Downloading separate debug info for /lib64/libboost_regex.so.1.78.0 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/ecbb0bd3a76b91d4668e8cbdaef774b2b347121c/debuginfo Downloading separate debug info for /lib64/libbz2.so.1 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f8237ff24b622047428b97b1486b393e3a9e38de/debuginfo Downloading separate debug info for /lib64/libpcre2-8.so.0 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/1ef098ba40305d86ec0cdae1e6aac188b1656550/debuginfo Downloading separate debug info for /lib64/libicudata.so.72 Downloading separate debug info for /lib64/libicui18n.so.72 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/13e5dae11fa14e4ac67e847e96ff38e07b8470a7/debuginfo Downloading separate debug info for /lib64/libicuuc.so.72 Downloading separate debug info for /lib64/libnghttp2.so.14 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/e10861ccd7fab76bc152927562f85635e27008f9/debuginfo Downloading separate debug info for /lib64/libidn2.so.0 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/c25f9a2ef9b6fb843df0fc37c6c8903af04cbbe1/debuginfo Downloading separate debug info for /lib64/libpsl.so.5 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/973e286449bb9a8694cdb165b372442956f8ded0/debuginfo Downloading separate debug info for /lib64/libssl.so.3 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f762bb3d54ccc26f5182b558e1dfde142d778f73/debuginfo Downloading separate debug info for /lib64/libcrypto.so.3 Downloading separate debug info for /lib64/libldap.so.2 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f28845a559a64c25c9febe01e95aadfef853e597/debuginfo Downloading separate debug info for /lib64/liblber.so.2 Downloading separate debug info for /lib64/libbrotlidec.so.1 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/b227ed45334b5f849f2585438a1b50e5437a6974/debuginfo Downloading separate debug info for /lib64/libunistring.so.2 Downloading separate debug info for /lib64/libevent-2.1.so.7 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/8e12d5357a281470bc5db4ef22e74d983bf2cd49/debuginfo Downloading separate debug info for /lib64/libsasl2.so.3 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/f88c0fc23af93f1bd86b93b6c8e2c9d72bbb98b7/debuginfo Downloading separate debug info for /lib64/libbrotlicommon.so.1 Downloading separate debug info for /lib64/libresolv.so.2 Downloading separate debug info for /lib64/libcrypt.so.2 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/ec8602178ec0a572468bc35fad45528176e55224/debuginfo [Detaching after vfork from child process 1201] [New Thread 0x7fffe69ff6c0 (LWP 1202)] [New Thread 0x7fffe61fe6c0 (LWP 1203)] [New Thread 0x7fffe59fd6c0 (LWP 1204)] [New Thread 0x7fffe51fc6c0 (LWP 1205)] [New Thread 0x7fffe49fb6c0 (LWP 1206)] [New Thread 0x7fffd7fff6c0 (LWP 1207)] [New Thread 0x7fffd77fe6c0 (LWP 1208)] [New Thread 0x7fffd6ffd6c0 (LWP 1209)] Reading symbols from /bin/ls... Downloading separate debug info for /usr/lib64/ossl-modules/legacy.so [New Thread 0x7fffd65fc6c0 (LWP 1210)] Downloading separate debug info for /usr/bin/ls [### ]Downloading separate debug info for /lib64/libnss_myhostname.so.2 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/e691afe9fc8f8d74f8ee0efcbee363eebe296b8e/debuginfo Downloading separate debug info for /lib64/libcap.so.2 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/2fe8ebb5811811046e04d8f4c000696cb9b24e24/debuginfo Downloading separate debug info for /lib64/libnss_mdns4_minimal.so.2 Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/fdb94adac135155164c94a95a342d03685b6804e/debuginfo Downloading separate debug info for /lib64/libnss_resolve.so.2 [Thread 0x7fffd65fc6c0 (LWP 1210) exited] Reading symbols from /home/kdudka/.cache/debuginfod_client/a146002a40b622e4d8420c5b96513f0784c8eb7b/debuginfo... Downloading separate debug info for /home/kdudka/.cache/debuginfod_client/a146002a40b622e4d8420c5b96513f0784c8eb7b/debuginfo Downloading separate debug info for /usr/lib64/gconv/UTF-32.so (gdb) quit Thread 1 "gdb" received signal SIGSEGV, Segmentation fault. ___pthread_rwlock_rdlock (rwlock=0x0) at pthread_rwlock_rdlock.c:26 Downloading source file /usr/src/debug/glibc-2.37-4.fc38.x86_64/nptl/pthread_rwlock_rdlock.c 26 int result = __pthread_rwlock_rdlock_full64 (rwlock, CLOCK_REALTIME, NULL); (gdb) bt #0 ___pthread_rwlock_rdlock (rwlock=0x0) at pthread_rwlock_rdlock.c:26 #1 0x00007ffff41c0f8d in CRYPTO_THREAD_read_lock (lock=<optimized out>) at crypto/threads_pthread.c:97 #2 0x00007ffff41b901f in ossl_lib_ctx_get_data (ctx=<optimized out>, index=1, meth=0x7ffff43ddaa0 <provider_store_method.lto_priv>) at crypto/context.c:413 #3 0x00007ffff41c659d in get_provider_store (libctx=<optimized out>) at crypto/provider_core.c:339 #4 ossl_provider_deregister_child_cb (handle=0x555556741220) at crypto/provider_core.c:1806 #5 0x00007ffff41b9f47 in ossl_provider_deinit_child (ctx=0x5555566fe840) at crypto/provider_child.c:281 #6 OSSL_LIB_CTX_free (ctx=0x5555566fe840) at crypto/context.c:284 #7 OSSL_LIB_CTX_free (ctx=0x5555566fe840) at crypto/context.c:277 #8 0x00007ffff4813bee in legacy_teardown (provctx=0x5555567414e0) at providers/legacyprov.c:168 #9 0x00007ffff41c69fb in ossl_provider_teardown (prov=0x555556741220) at crypto/provider_core.c:1558 #10 ossl_provider_free (prov=0x555556741220) at crypto/provider_core.c:696 #11 0x00007ffff41904bc in ossl_provider_free (prov=<optimized out>) at crypto/provider_core.c:681 #12 evp_cipher_free_int (cipher=0x555556783fe0) at crypto/evp/evp_enc.c:1651 #13 EVP_CIPHER_free (cipher=0x555556783fe0) at crypto/evp/evp_enc.c:1666 #14 0x00007ffff4996abd in ssl_evp_cipher_free (cipher=0x555556783fe0) at ssl/ssl_lib.c:5967 #15 ssl_evp_cipher_free (cipher=0x555556783fe0) at ssl/ssl_lib.c:5957 #16 SSL_CTX_free (a=0x55555678d610) at ssl/ssl_lib.c:3492 #17 SSL_CTX_free (a=0x55555678d610) at ssl/ssl_lib.c:3429 #18 0x00007ffff6da29b7 in ossl_close (cf=0x555556742ef8, data=0x55555674ab88) at vtls/openssl.c:1901 #19 0x00007ffff6daa737 in cf_close (cf=0x555556742ef8, data=0x55555674ab88) at vtls/vtls.c:1450 #20 0x00007ffff6daa991 in ssl_cf_destroy (cf=0x555556742ef8, data=0x55555674ab88) at vtls/vtls.c:1510 #21 0x00007ffff6d0f699 in Curl_conn_cf_discard_chain (pcf=0x555556744c70, data=0x55555674ab88) at cfilters.c:165 #22 0x00007ffff6d0b683 in cf_hc_close (cf=0x555556744c68, data=0x55555674ab88) at cf-https-connect.c:436 #23 0x00007ffff6d0f780 in Curl_conn_close (data=0x55555674ab88, index=0) at cfilters.c:186 #24 0x00007ffff6d8ebeb in conn_shutdown (data=0x55555674ab88) at url.c:685 #25 0x00007ffff6d8f1fb in Curl_disconnect (data=0x55555674ab88, conn=0x555556742ff8, dead_connection=false) at url.c:796 #26 0x00007ffff6d12039 in Curl_conncache_close_all_connections (connc=0x555556746b60) at conncache.c:532 #27 0x00007ffff6d62eb3 in curl_multi_cleanup (multi=0x555556746a38) at multi.c:2831 #28 0x00007ffff7bc154e in debuginfod_end (client=0x555556746360) at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1892 #29 debuginfod_end (client=0x555556746360) at /usr/src/debug/elfutils-0.189-2.fc38.x86_64/debuginfod/debuginfod-client.c:1887 #30 0x000055555589a9c5 in debuginfod_client_deleter::operator() (c=<optimized out>, this=<optimized out>) at ../../gdb/debuginfod-support.c:105 #31 std::unique_ptr<debuginfod_client, debuginfod_client_deleter>::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/13/bits/unique_ptr.h:404 #32 0x00007ffff70621a6 in __run_exit_handlers (status=0, listp=<optimized out>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:111 #33 0x00007ffff70622ee in __GI_exit (status=<optimized out>) at exit.c:141 #34 0x0000555555c3009a in quit_force (exit_arg=<optimized out>, from_tty=from_tty@entry=1) at ../../gdb/top.c:1894 #35 0x000055555582914c in quit_command (args=0x0, from_tty=1) at ../../gdb/cli/cli-cmds.c:506 #36 0x0000555555831065 in cmd_func (cmd=<optimized out>, args=<optimized out>, from_tty=<optimized out>) at ../../gdb/cli/cli-decode.c:2543 #37 0x0000555555c312a5 in execute_command (p=<optimized out>, from_tty=<optimized out>) at ../../gdb/top.c:688 #38 0x0000555555931f11 in gdb_rl_callback_handler (rl=0x0) at ../../gdb/event-top.c:247 #39 0x00007ffff7d7b0cd in rl_callback_read_char () at ../callback.c:299 #40 0x0000555555932076 in gdb_rl_callback_read_char_wrapper_noexcept () at ../../gdb/event-top.c:189 #41 0x0000555555932228 in gdb_rl_callback_read_char_wrapper (client_data=<optimized out>) at ../../gdb/event-top.c:221 #42 0x0000555555930f90 in stdin_event_handler (error=<optimized out>, client_data=0x5555563e1730) at ../../gdb/event-top.c:544 #43 0x0000555555e9097e in gdb_wait_for_event (block=block@entry=1) at ../../gdbsupport/event-loop.cc:716 #44 0x0000555555e91532 in gdb_do_one_event (mstimeout=mstimeout@entry=-1) at ../../gdbsupport/event-loop.cc:264 #45 0x0000555555a18482 in start_event_loop () at ../../gdb/main.c:411 #46 captured_command_loop () at ../../gdb/main.c:471 #47 0x0000555555a1a5f5 in captured_main (data=0x7fffffffe130) at ../../gdb/main.c:1330 #48 gdb_main (args=args@entry=0x7fffffffe160) at ../../gdb/main.c:1345 #49 0x00005555557079ef in main (argc=<optimized out>, argv=<optimized out>) at ../../gdb/gdb.c:40
The issue seems to be that both libcrypto.so and gdb register atexit() handlers, which are later executed in the unfavorable order: 1. OpenSSL's exit handler calls OPENSSL_cleanup(), which wipes global data of OpenSSL. 2. gdb's exit handler calls debuginfod_end() from libdebuginfod.so, which calls curl_multi_cleanup() from libcurl.so, which in turn calls SSL_CTX_free() when OpenSSL is already deinitialized. I do not think there is anything that libcurl could do better. Switching back to gdb...
(In reply to Kamil Dudka from comment #6) > The issue seems to be that both libcrypto.so and gdb register atexit() > handlers, which are later executed in the unfavorable order: > > 1. OpenSSL's exit handler calls OPENSSL_cleanup(), which wipes global data > of OpenSSL. > 2. gdb's exit handler calls debuginfod_end() from libdebuginfod.so, which > calls curl_multi_cleanup() from libcurl.so, which in turn calls > SSL_CTX_free() when OpenSSL is already deinitialized. > > I do not think there is anything that libcurl could do better. Switching > back to gdb... libcrypto's exit handler is registered during curl_global_init() and the error happens only because curl_multi_cleanup is called during exit. AFAICT it should be ok to call curl_multi_cleanup during exit. I can't find anything in the docs to suggest otherwise, and things work as expected when we aren't using these openssl legacy settings. This unfavorable order of handlers still looks like a libcurl issue. gdb is using libcurl according to spec.
I think it is a bad idea to call curl_multi_cleanup() from an atexit() handler, where one can hardly assume anything about the global state of the underlying libraries.
The problem is that the atexit handlers are called in refers order. The unique_ptr deleter for the debuginfod_client is registered before the debuginfod_client is created for the first time. This is how debuginfod_begin works (the first time it is called): static pthread_once_t init_control = PTHREAD_ONCE_INIT; static void libcurl_init(void) { curl_global_init(CURL_GLOBAL_DEFAULT); } debuginfod_client * debuginfod_begin (void) { /* Initialize libcurl lazily, but only once. */ pthread_once (&init_control, libcurl_init); ... So I think this code in gdb/debuginfod-support.c registers its at_exit handler for the deleter first: /* Deleter for a debuginfod_client. */ struct debuginfod_client_deleter { void operator() (debuginfod_client *c) { debuginfod_end (c); } }; using debuginfod_client_up = std::unique_ptr<debuginfod_client, debuginfod_client_deleter>; So debuginfod_end will be called after libcrypto's exit handler has been called. I don't know how to switch the registration order of the at_exit handlers.
I think the patch below will ensure that the debuginfod_client object is cleaned up (debuginfod_end is called) before the at_exit handlers are run. I'm currently setting up an environment to test this, but thought I'd post this here in case anyone else was also looking at this. diff --git a/gdb/debuginfod-support.c b/gdb/debuginfod-support.c index 5853f420a18..c59f0f21956 100644 --- a/gdb/debuginfod-support.c +++ b/gdb/debuginfod-support.c @@ -180,20 +180,83 @@ progressfn (debuginfod_client *c, long cur, long total) return 0; } -static debuginfod_client * -get_debuginfod_client () +/* A class to manage setting up (and cleaning up) a debuginfod_client + object. GDB should only have a single debuginfod_client object, so we + should only create a single instance of this class. */ + +struct debuginfod_client_manager { - static debuginfod_client_up global_client; + /* Return a pointer to the managed debuginfod_client object, + initialising it first if needed. */ - if (global_client == nullptr) - { - global_client.reset (debuginfod_begin ()); + debuginfod_client *get () + { + if (m_global_client == nullptr) + this->init (); + return m_global_client.get (); + } - if (global_client != nullptr) - debuginfod_set_progressfn (global_client.get (), progressfn); - } +private: + + /* Callback used by GDB's final cleanup mechanism. Cleans up the + debuginfod_client object being managed by ARG, which should be a + debuginfod_client_manager pointer. */ + + static void cleanup_debuginfod_client (void *arg) + { + debuginfod_client_manager *m = (debuginfod_client_manager *) arg; + m->cleanup (); + } + + /* Initialise the managed debuginfod_client object and register with + GDB's global cleanup mechanism to ensure that the object is cleaned up + correctly. */ + void init () + { + gdb_assert (m_global_client == nullptr); + + debuginfod_client *client = debuginfod_begin (); + + if (client != nullptr) + { + /* Debuginfod (or the libraries it uses) make use of at_exit for + final cleanup. Meanwhile, GDB makes use of C++ object + destruction to cleanup global variables. If we make + m_global_client a global and rely on the C++ finalisation to + call the unique_ptr destructor then this is done too late, by + this point the at_exit handlers have already run and so the + debuginfod_client object will already be partially cleaned up. + + Instead, register a cleanup with GDB's final cleanup mechanism. + This is run by GDB before calling exit, which means we will + cleanup the debuginfod_client before calling exit, and so before + calling the at_exit handlers. */ + make_final_cleanup (cleanup_debuginfod_client, this); + debuginfod_set_progressfn (client, progressfn); + m_global_client.reset (client); + } + } + + /* The managed debuginfod_client object. */ + debuginfod_client_up m_global_client; +}; + +/* Return a pointer to the single global debuginfod_client, initialising it + first if needed. */ - return global_client.get (); +static debuginfod_client * +get_debuginfod_client () +{ + static debuginfod_client_manager global_client_manager; + return global_client_manager.get (); } /* Check if debuginfod is enabled. If configured to do so, ask the user
I've sent a version of the above patch to the GDB mailing list: https://sourceware.org/pipermail/gdb-patches/2023-May/199779.html for consideration.
Upstream commit f3eee5861743d635ea3a5be7eae3115f85ec9dae fixes this issue. I've opened pull requests #87 and #88 to merge backport this fix to rawhide and f38 respectively.
FEDORA-2023-42bcd5dc5d has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-42bcd5dc5d
FEDORA-2023-42bcd5dc5d has been pushed to the Fedora 38 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-42bcd5dc5d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-42bcd5dc5d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-42bcd5dc5d has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.