Bug 1973026

Summary: Thread debugging broken in glibc-2.33.9000-18.fc35
Product: [Fedora] Fedora Reporter: Kevin Buettner <kevinb>
Component: glibcAssignee: Florian Weimer <fweimer>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: aoliva, arjun.is, codonell, dj, fweimer, law, mcermak, mfabian, pfrankli, rth, sipoyare
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glibc-2.33.9000-20.fc35 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-17 15:59:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kevin Buettner 2021-06-17 06:43:29 UTC
Description of problem:

I updated one of my x86-64 rawhide VMs and then installed the following packages:

glibc-all-langpacks-2.33.9000-18.fc35.x86_64
glibc-common-2.33.9000-18.fc35.x86_64
glibc-langpack-en-2.33.9000-18.fc35.x86_64
glibc-headers-x86-2.33.9000-18.fc35.noarch
glibc-devel-2.33.9000-18.fc35.x86_64
glibc-doc-2.33.9000-18.fc35.noarch
glibc-debuginfo-2.33.9000-18.fc35.x86_64
glibc-2.33.9000-18.fc35.i686
glibc-devel-2.33.9000-18.fc35.i686
glibc-static-2.33.9000-18.fc35.x86_64
glibc-common-debuginfo-2.33.9000-18.fc35.x86_64
glibc-2.33.9000-18.fc35.x86_64

I fetched the RPMs for those packages from this page:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1772416

When I attempt to debug one of the gdb.threads test cases, I see:

[kev@rawhide-1 gdb]$ ./gdb -q testsuite/outputs/gdb.threads/tls/tls 
Reading symbols from testsuite/outputs/gdb.threads/tls/tls...
(gdb) start
Temporary breakpoint 1 at 0x4015fd: file /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c, line 231.
Starting program: /mesquite2/sourceware-git/rawhide-glibc234/bld/gdb/testsuite/outputs/gdb.threads/tls/tls 

Temporary breakpoint 1, main () at /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c:231
231	   do_pass ();
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7fcc090  0x00007ffff7ff0fd6  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7e2e2f0  0x00007ffff7f27eb2  Yes         /lib64/libstdc++.so.6
0x00007ffff7cc1390  0x00007ffff7d30fd8  Yes         /lib64/libm.so.6
0x00007ffff7c9a5f0  0x00007ffff7cab955  Yes         /lib64/libgcc_s.so.1
0x00007ffff7acd700  0x00007ffff7c31d5d  Yes (*)     /lib64/libc.so.6
(*): Shared library is missing debugging information.

Note that the output from "info shared" indicates that /lib64/libc.so.6 is missing debugging information.  Also, there should be a libthread_db initialization message; it is missing from the above session.

Poking around on the filesystem, I find:

[kev@rawhide-1 gdb]$ ls -l {,/usr/lib/debug}/usr/lib64/libc[.-]*
-rw-r--r-- 1 root root 15319958 Jun 16 01:20 /usr/lib64/libc.a
-rw-r--r-- 1 root root      253 Jun 16 01:16 /usr/lib64/libc.so
-rwxr-xr-x 1 root root  2023736 Jun 16 01:20 /usr/lib64/libc.so.6
-rw-r--r-- 1 root root  6936944 Jun 16 01:20 /usr/lib/debug/usr/lib64/libc.so.6-2.33.9000-18.fc35.x86_64.debug

Now, let's look at the build id using eu-unstrip:

[kev@rawhide-1 gdb]$ eu-unstrip --list-only -e /usr/lib64/libc.so.6 
0+0x1f30d0 47b3653bd8d4a01f90cfcfdb63f57b2451bb6572@0x3b0 /usr/lib64/libc.so.6 - 
[kev@rawhide-1 gdb]$ eu-unstrip --list-only -e /usr/lib/debug/usr/lib64/libc.so.6-2.33.9000-18.fc35.x86_64.debug
0+0x1f30d0 dd8443cd4b5b0eb7191cbf884725c3706bebc519@0x3b0 /usr/lib/debug/usr/lib64/libc.so.6-2.33.9000-18.fc35.x86_64.debug .

It's 47b3653bd8d4a01f90cfcfdb63f57b2451bb6572 for the library
and dd8443cd4b5b0eb7191cbf884725c3706bebc519 for the .debug file.

Note that the build ids don't match.

Version-Release number of selected component (if applicable):

See above.

How reproducible:

Always.

Steps to Reproduce:

See above.

Actual results:

See above.

Expected results:

On another rawhide machine with glibc-2.33.9000-12.fc35.x86_64 (and related packages) installed, I see:

[kev@rawhide-glibc-test ~]$ ls -l {,/usr/lib/debug}/usr/lib64/libc[.-]*
-rwxr-xr-x 1 root root  2054848 Jun  3 05:13 /usr/lib64/libc-2.33.9000.so
-rw-r--r-- 1 root root 15287622 Jun  3 05:13 /usr/lib64/libc.a
-rw-r--r-- 1 root root      253 Jun  3 05:07 /usr/lib64/libc.so
lrwxrwxrwx 1 root root       17 Jun  3 05:08 /usr/lib64/libc.so.6 -> libc-2.33.9000.so
-rw-r--r-- 1 root root  6935592 Jun  3 05:13 /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug
-rw-r--r-- 1 root root 36054446 Apr 15  2020 /usr/lib/debug/usr/lib64/libc.a

eu-unstrip shows matching build ids for libc and the debuginfo file:

[kev@rawhide-glibc-test ~]$ eu-unstrip --list-only -e /usr/lib64/libc-2.33.9000.so
0+0x1f30b0 8502c0eadde138b04c06d3f8b311c69d6be65669@0x3b0 /usr/lib64/libc-2.33.9000.so /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug 
[kev@rawhide-glibc-test ~]$ eu-unstrip --list-only -e /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug
0+0x1f30b0 8502c0eadde138b04c06d3f8b311c69d6be65669@0x3b0 /usr/lib/debug/usr/lib64/libc-2.33.9000.so-2.33.9000-12.fc35.x86_64.debug . 

Also gdb shows that it's loaded / initialized libthread_db:

[kev@rawhide-glibc-test gdb]$ ./gdb -q testsuite/outputs/gdb.threads/tls/tls
Reading symbols from testsuite/outputs/gdb.threads/tls/tls...
(gdb) start
Temporary breakpoint 1 at 0x4015fd: file /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c, line 231.
Starting program: /mesquite2/sourceware-git/rawhide-glibc234/bld/gdb/testsuite/outputs/gdb.threads/tls/tls 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Temporary breakpoint 1, main () at /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.c:231
231	   do_pass ();
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7fcc090  0x00007ffff7ff0fc6  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7e2e2f0  0x00007ffff7f27eb2  Yes         /lib64/libstdc++.so.6
0x00007ffff7cc1390  0x00007ffff7d30fd8  Yes         /lib64/libm.so.6
0x00007ffff7c9a5f0  0x00007ffff7cab955  Yes         /lib64/libgcc_s.so.1
0x00007ffff7acd700  0x00007ffff7c31e9d  Yes         /lib64/libc.so.6

Note this gdb session printed messages regarding libthread_db; these messages are missing when debugging on the machine w/ glibc-2.33.9000-18.fc35 installed.

Also note that "info shared" shows that the symbols are loaded and has no asterisk indicating missing debug info.

Comment 1 Kevin Buettner 2021-06-17 06:52:00 UTC
In case the original report isn't convincing enough...

[kev@rawhide-glibc-test gdb]$ rpm -q -a | grep -e ^glibc-2.33 | grep x86
glibc-2.33.9000-12.fc35.x86_64
[kev@rawhide-glibc-test gdb]$ make check TESTS=gdb.threads/tls.exp
...
Running /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.exp ...

		=== gdb Summary ===

# of expected passes		77
# of known failures		1
...

VERSUS:

[kev@rawhide-1 gdb]$ rpm -q -a | grep -e ^glibc-2.33 | grep x86
glibc-2.33.9000-18.fc35.x86_64
[kev@rawhide-1 gdb]$ make check TESTS=gdb.threads/tls.exp
...
Running /ironwood1/sourceware-git/rawhide-glibc234/bld/../../worktree-glibc234/gdb/testsuite/gdb.threads/tls.exp ...
FAIL: gdb.threads/tls.exp: at least one th in spin while stopped at first th
FAIL: gdb.threads/tls.exp: first thread local storage
FAIL: gdb.threads/tls.exp: first get symbol value without frame
FAIL: gdb.threads/tls.exp: first another thread local storage
FAIL: gdb.threads/tls.exp: at least one th in spin while stopped at second th
FAIL: gdb.threads/tls.exp: second thread local storage
FAIL: gdb.threads/tls.exp: second get symbol value without frame
FAIL: gdb.threads/tls.exp: second another thread local storage
FAIL: gdb.threads/tls.exp: at least one th in spin while stopped at third th
FAIL: gdb.threads/tls.exp: third thread local storage
FAIL: gdb.threads/tls.exp: third get symbol value without frame
FAIL: gdb.threads/tls.exp: third another thread local storage
FAIL: gdb.threads/tls.exp: get number of threads
FAIL: gdb.threads/tls.exp: no thread backtrace reported spin (vsyscall kernel problem?)
FAIL: gdb.threads/tls.exp: mess at end
FAIL: gdb.threads/tls.exp: p a_thread_local
FAIL: gdb.threads/tls.exp: p file2_thread_local
FAIL: gdb.threads/tls.exp: p a_thread_local second time

		=== gdb Summary ===

# of expected passes		25
# of unexpected failures	18
# of known failures		1

Comment 2 Florian Weimer 2021-06-17 08:20:48 UTC
Okay, my attempt with a minimal symtab clearly does not work. I'm going to revert this. Sorry about that.

However, libthread_db should be loaded even if glibc-debuginfo is not installed. Would it possible to change GDB to do that?

Comment 3 Fedora Update System 2021-06-17 10:49:30 UTC
FEDORA-2021-783138c496 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 4 Florian Weimer 2021-06-17 15:15:55 UTC
So it seems that pthread_create is actually visible to GDB without debuginfo (although I added it to .symtab as well), but we have hidden a bunch of stuff that libthread_db needs and can't see without at least a symbol table. I will investigate moving those symbols to .dynsym and if that enables libthread_db without debuginfo.

Comment 5 Kevin Buettner 2021-06-17 15:35:28 UTC
(In reply to Florian Weimer from comment #2)

> However, libthread_db should be loaded even if glibc-debuginfo is not
> installed. Would it possible to change GDB to do that?

GDB used to work without glibc's debuginfo present.  I'm not sure when or why this changed, but I'll investigate.

Comment 6 Kevin Buettner 2021-06-17 15:59:02 UTC
I've tested glibc-2.33.9000-21.fc35. Thread debugging is working again!

Comment 7 Fedora Update System 2021-06-17 20:58:30 UTC
FEDORA-2021-72b593648e has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.