Bug 1960867

Summary: thread debugging broken with gdb
Product: [Fedora] Fedora Reporter: Avi Kivity <avi.kivity>
Component: glibcAssignee: Carlos O'Donell <codonell>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 34CC: aoliva, arjun.is, codonell, dj, dns.botond, fedoraproject, fweimer, jan.kratochvil, keiths, kevinb, law, mcermak, mfabian, pfrankli, pmuldoon, roberttucker, rth, sergiodj, sipoyare
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-21 03:02:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Avi Kivity 2021-05-15 17:55:55 UTC
Description of problem:

Trying to debug a threaded program fails, because gdb can't find libthread_db.


Version-Release number of selected component (if applicable):


gdb-10.1-14.fc34.x86_64

How reproducible:


Steps to Reproduce:
1. compile an empty program (just main) with g++ -g -pthread test.cc
2. gdb a.out
3. issue the command 'start'

Actual results:

warning: Expected absolute pathname for libpthread in the inferior, but got .gnu_debugdata for /lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Accessing thread-local variables will not work.

Expected results:

No such warning.

Additional info:


Observed with g++ and clang++, and with ld.bfd and gold.

Comment 1 Robert Tucker 2021-05-29 07:32:53 UTC
Also getting this:

(gdb) run
Starting program: /usr/bin/vlc 
warning: Expected absolute pathname for libpthread in the inferior, but got .gnu_debugdata for /lib64/libpthread.so.0.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
VLC media player 4.0.0-dev Otto Chriek (revision 4.0.0-dev-15406-g183c8d9971)
[New LWP 14303]
[New LWP 14304]

Comment 2 Botond Dénes 2021-06-02 12:43:34 UTC
I have a variant of this:

Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
thread_db_load_search returning 0
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib64/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
warning: Expected absolute pathname for libpthread in the inferior, but got .gnu_debugdata for /lib64/libpthread.so.0.
Trying host libthread_db library: /lib64/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
thread_db_load_search returning 0


Looking at the gdb source, the error "warning: Expected absolute pathname for libpthread in the inferior, but got .gnu_debugdata for /lib64/libpthread.so.0." comes from a function which just tries to find the path to load libthread-db from. The other error "td_ta_new failed: application not linked with libthread" comes when initializing libthread-db. So it looks like there are two problems (which might be connected): an unexpected path to libpthread.so and libthread_db.so failing the initialization after having been found and loaded.

Comment 3 Botond Dénes 2021-06-02 15:31:30 UTC
For me installing glibc debug info made the problem go away for now:

sudo dnf debuginfo-install glibc-2.32-6.fc33.x86_64

Comment 4 Kevin Buettner 2021-07-12 03:04:42 UTC
It turns out that this is a problem in both Fedora 33 and Fedora 34.  I've investigated and have found that the symbol 'nptl_version' is not present in .symtab or .dynsym in libpthread (or even libc on F34). GDB (on behalf of td_ta_new in libthread_db) needs to be able to find the address of nptl_version in order for libthread_db to be loaded / initialized.

To verify this, run the following commands on the specified Fedora releases:

Fedora 32:

[kev@f32-1 ~]$ objdump -t -T /lib64/libpthread-2.31.so | grep nptl_version
0000000000017037 l     O .rodata	0000000000000005              nptl_version

This shows that nptl_version is present in either .symtab (-t switch) or .dynsym (-T switch) in the libpthread library.

Fedora 33:

[kev@f33-1 ~]$ objdump -t -T /lib64/libpthread-2.32.so | grep nptl_version
[kev@f33-1 ~]$ 

I.e. no nptl_version symbol found.

Fedora 34:

[kev@f34-1 ~]$ objdump -t -T /lib64/libpthread-2.33.so | grep nptl_version
[kev@f34-1 ~]$ objdump -t -T /lib64/libc-2.33.so | grep nptl_version

Likewise, nptl_version wasn't found in Fedora 34.

Rawhide (with glibc-2.33.9000-37 installed):

[kev@rawhide-1 ~]$ objdump -t -T /lib64/libc.so.6 | grep nptl_version
00000000001b5688 g     O .rodata	000000000000000a              __nptl_version
00000000001b5688 g    DO .rodata	000000000000000a  GLIBC_PRIVATE __nptl_version


Note that the symbol name has changed somewhat, but it's still there. I've tested GDB and have found that it works as expected without debuginfo packages installed (and also with debuginfod disabled, plus cached debuginfod packages deleted).

So... Fedora 33 and Fedora 34 need fixing.

Comment 5 Florian Weimer 2021-07-13 17:02:03 UTC
Kevin, would you please test the latest Fedora 33 and 34 builds?

Fedora 33: glibc-2.32-10.fc33
Fedora 34: glibc-2.33-20.fc34

I expect the issue has been fixed there under bug 1975895. Thanks.

Comment 6 Kevin Buettner 2021-07-21 03:02:01 UTC
I've tested gdb.threads/tls.exp and gdb.threads/linux-dp.exp on Fedora 33 and Fedora 34. On both OSes, I removed the glibc debuginfo packages. No failures from any of the tests.

I've also checked to see whether nptl_version is present in libpthread as described in Comment 4. I found the following (which is good):

Fedora 33:

[kev@f33-1 gdb]$ objdump -t -T /lib64/libpthread-2.32.so | grep nptl_version
0000000000017037 l     O .rodata	0000000000000005              nptl_version

Fedora 34:

[kev@f34-1 gdb]$ objdump -t -T /lib64/libpthread-2.33.so | grep nptl_version
0000000000016037 l     O .rodata	0000000000000005              nptl_version

So... it looks fixed to me.  I'm closing it.