Bug 1986018

Summary: fedora 34+ debugsource files lack assembler .S files
Product: Red Hat Enterprise Linux 9 Reporter: Frank Ch. Eigler <fche>
Component: binutilsAssignee: Nick Clifton <nickc>
binutils sub component: system-version QA Contact: Miloš Prchlík <mprchlik>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: low    
Priority: unspecified CC: aoliva, arjun, bstinson, codonell, dj, dvlasenk, extras-qa, fweimer, jakub, jwboyer, law, mcermak, mfabian, mjw, mprchlik, nickc, ohudlick, pfrankli, rth, sipoyare, tschelle
Version: CentOS StreamKeywords: Bugfix, Triaged
Target Milestone: betaFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: binutils-2.35.2-8.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1966987 Environment:
Last Closed: 2021-12-07 21:35:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1966987    
Bug Blocks:    

Description Frank Ch. Eigler 2021-07-26 14:13:51 UTC
+++ This bug was initially created as a clone of Bug #1966987 +++

The build process for glibc includes a mixture of C and assembly code.  The DWARF debuginfo (post-debugedit) includes file name references such as

/usr/src/debug/glibc-2.33.9000-73-ge9e7f24543/sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S

However, the -debugsource subrpm does not include any .s / .S files at all.  This makes debugging a little big harder, and makes the fedora debuginfod a little bit sad to report 404s to users.

--- Additional comment from Florian Weimer on 2021-06-02 10:04:48 UTC ---

Thanks for the report. I think this needs to be fixed in debugedit. We no longer do our own thing in glibc.

--- Additional comment from Frank Ch. Eigler on 2021-06-02 10:50:03 UTC ---

It looks as though something changed in the f34 build process.

Looking at other RPMs such as the recently built openssl-1.1.1k in koji, builds up to and including fc33 include .s files in the -debugsource.  f34 and above do not.

And ditto for GLIBC: 2.32-6.fc33 includes .S files in -debugsource.

--- Additional comment from Mark Wielaard on 2021-06-02 10:54:14 UTC ---

(In reply to Frank Ch. Eigler from comment #2)
> It looks as though something changed in the f34 build process.

That is surprising, is the same true for f35?
In f35 rpm switched to using external debugedit find-debuginfo[.sh].

But f34 should use the same rpm find-debuginfo.sh as earlier versions.

--- Additional comment from Frank Ch. Eigler on 2021-06-02 12:18:02 UTC ---

From my quick sampling tests, debugsource files
contain .S files <= fc33
lack .S files >= fc34

--- Additional comment from Frank Ch. Eigler on 2021-06-02 13:46:30 UTC ---

hand-rebuilding rawhide glibc on f33 (fedpkg local) results in a glibc-debugsource-2.33.9000-11.fc35.x86_64.rpm file that does contain .S files.

--- Additional comment from Frank Ch. Eigler on 2021-06-02 21:16:42 UTC ---

More data: gcc -gdwarf5 defaulting appears to be the driving difference.  Even for a noop assembler file

foo.s:

   main:
       nop

gcc -gdwarf-5 -c foo.s   # results in an .o with dwarf5 line records
gcc -gdwarf-5 -c foo.s   # results in an .o with old school records

and "debugedit -l FILE foo.o" only works with the latter.  Continuing to investigate.

--- Additional comment from Frank Ch. Eigler on 2021-06-02 21:17:04 UTC ---

gcc -gdwarf-5 -c foo.s   # results in an .o with dwarf5 line records
gcc -gdwarf-4 -c foo.s   # results in an .o with old school records

--- Additional comment from Frank Ch. Eigler on 2021-06-02 22:49:01 UTC ---

Yeah it appears as though debugedit just can't extract source file names from -gdwarf-5 files, not just .s but also .c:

% echo 'int main () {}' > foo.c
% gcc -gdwarf-5 foo.c
% debugedit -l foo.list a.out
% cat foo.list
[empty]

--- Additional comment from Mark Wielaard on 2021-06-22 17:29:14 UTC ---

This should at least solve the issue of comment 8. Without giving -d /foo/build/bar debugedit wouldn't actually output any file from the line number table for DWARF5.

diff --git a/tools/debugedit.c b/tools/debugedit.c
index c6975b2..75803a2 100644
--- a/tools/debugedit.c
+++ b/tools/debugedit.c
@@ -1826,8 +1826,8 @@ read_dwarf5_line_entries (DSO *dso, unsigned char **ptrp,
   /* file_names_count */
   unsigned entry_count = read_uleb128 (*ptrp);
 
-  bool collecting_dirs = dest_dir && phase == 0 && *dirs == NULL;
-  bool writing_files = dest_dir && phase == 0 && *dirs != NULL;
+  bool collecting_dirs = phase == 0 && *dirs == NULL;
+  bool writing_files = phase == 0 && *dirs != NULL;
   if (collecting_dirs)
     {
       *ndir = entry_count;
@@ -1857,15 +1857,19 @@ read_dwarf5_line_entries (DSO *dso, unsigned char **ptrp,
                {
                case DW_FORM_strp:
                case DW_FORM_line_strp:
-                 if (dest_dir && phase == 0)
+                 if (phase == 0)
                    {
                      size_t idx = do_read_32_relocated (*ptrp);
-                     if (record_file_string_entry_idx (line_strp, dso, idx))
+                     if (dest_dir)
                        {
-                         if (line_strp)
-                           need_line_strp_update = true;
-                         else
-                           need_strp_update = true;
+                         if (record_file_string_entry_idx (line_strp,
+                                                           dso, idx))
+                           {
+                             if (line_strp)
+                               need_line_strp_update = true;
+                             else
+                               need_strp_update = true;
+                           }
                        }
                      handled_strp = true;
                      if (collecting_dirs || writing_files)
@@ -1951,7 +1955,7 @@ read_dwarf5_line_entries (DSO *dso, unsigned char **ptrp,
       if (writing_files)
        {
          char *comp_dir = (*dirs)[0];
-         size_t comp_dir_len = strlen(comp_dir);
+         size_t comp_dir_len = !comp_dir ? 0 : strlen(comp_dir);
          size_t file_len = strlen (file);
          size_t dir_len = strlen (dir);
 

I am not sure this also solves the original issue because during packaging debugedit should always be invoked with --base ... and --dest ...

--- Additional comment from Mark Wielaard on 2021-07-26 12:22:36 UTC ---

As far as I can see there simply is no debuginfo for these .S files:

# eu-readelf -S ./glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/csu/sysdep.o
There are 11 section headers, starting at offset 0x190:

Section Headers:
[Nr] Name                 Type         Addr             Off      Size     ES Flags Lk Inf Al
[ 0]                      NULL         0000000000000000 00000000 00000000  0        0   0  0
[ 1] .text                PROGBITS     0000000000000000 00000040 00000000  0 AX     0   0  1
[ 2] .data                PROGBITS     0000000000000000 00000040 00000000  0 WA     0   0  1
[ 3] .bss                 NOBITS       0000000000000000 00000040 00000000  0 WA     0   0  1
[ 4] .note.gnu.property   NOTE         0000000000000000 00000040 00000050  0 A      0   0  8
[ 5] .note.GNU-stack      PROGBITS     0000000000000000 00000090 00000000  0        0   0  1
[ 6] .gnu.build.attributes NOTE         0000000000000000 00000090 00000024  0        0   0  4
[ 7] .rela.gnu.build.attributes RELA         0000000000000000 000000f0 00000030 24 I      8   6  8
[ 8] .symtab              SYMTAB       0000000000000000 000000b8 00000030 24        9   2  8
[ 9] .strtab              STRTAB       0000000000000000 000000e8 00000001  0        0   0  1
[10] .shstrtab            STRTAB       0000000000000000 00000120 0000006a  0        0   0  1

So nothing for debugedit to extract.

This might be a gcc (or binutils gas?) bug because it does look like -g is being passed when building the file:

gcc ../sysdeps/unix/sysv/linux/x86_64/sysdep.S -c     -I../include -I/root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/csu  -I/root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux  -I../sysdeps/unix/sysv/linux/x86_64/64  -I../sysdeps/unix/sysv/linux/x86_64  -I../sysdeps/unix/sysv/linux/x86/include -I../sysdeps/unix/sysv/linux/x86  -I../sysdeps/x86/nptl  -I../sysdeps/unix/sysv/linux/wordsize-64  -I../sysdeps/x86_64/nptl  -I../sysdeps/unix/sysv/linux/include -I../sysdeps/unix/sysv/linux  -I../sysdeps/nptl  -I../sysdeps/pthread  -I../sysdeps/gnu  -I../sysdeps/unix/inet  -I../sysdeps/unix/sysv  -I../sysdeps/unix/x86_64  -I../sysdeps/unix  -I../sysdeps/posix  -I../sysdeps/x86_64/64  -I../sysdeps/x86_64/fpu/multiarch  -I../sysdeps/x86_64/fpu  -I../sysdeps/x86/fpu  -I../sysdeps/x86_64/multiarch  -I../sysdeps/x86_64  -I../sysdeps/x86/include -I../sysdeps/x86  -I../sysdeps/ieee754/float128  -I../sysdeps/ieee754/ldbl-96/include -I../sysdeps/ieee754/ldbl-96  -I../sysdeps/ieee754/dbl-64  -I../sysdeps/ieee754/flt-32  -I../sysdeps/wordsize-64  -I../sysdeps/ieee754  -I../sysdeps/generic  -I.. -I../libio -I. -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/11/include -isystem /usr/include -D_LIBC_REENTRANT -include /root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/libc-modules.h -DMODULE_NAME=libc -include ../include/libc-symbols.h  -DPIC -DSHARED     -DTOP_NAMESPACE=glibc -DASSEMBLER -I/root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/csu/. -fcf-protection -include cet.h -g -Wa,--generate-missing-build-notes=yes -Werror=undef -Wa,--noexecstack   -o /root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/csu/sysdep.os -MD -MP -MF /root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/csu/sysdep.os.dt -MT /root/glibc/glibc-2.33.9000-984-gddcc612ce9/build-x86_64-redhat-linux/csu/sysdep.os

--- Additional comment from Mark Wielaard on 2021-07-26 12:46:21 UTC ---

Strangely when removing -DPIC from the command line makes the .debug_* sections appear again... I don't know why.

--- Additional comment from Siddhesh Poyarekar on 2021-07-26 12:53:11 UTC ---

(In reply to Mark Wielaard from comment #11)
> Strangely when removing -DPIC from the command line makes the .debug_*
> sections appear again... I don't know why.

The TU is empty if PIC is defined:

~~~~
#ifndef PIC                                                                     
                                                                                
/* The syscall stubs jump here when they detect an error.                       
   The code for Linux is almost identical to the canonical Unix                 
   code, except that the error number in %rax is negated.  */                   
                                                                                
#undef CALL_MCOUNT                                                              
#define CALL_MCOUNT /* Don't insert the profiling call, it clobbers %rax.  */   
                                                                                
        .text                                                                   
ENTRY (__syscall_error)                                                         
        neg %RAX_LP                                                             
                                                                                
#define __syscall_error __syscall_error_1                                       
#include <sysdeps/unix/x86_64/sysdep.S>                                         
                                                                                
#endif  /* !PIC */               
~~~~

--- Additional comment from Mark Wielaard on 2021-07-26 13:50:40 UTC ---

So that explains the sysdep.o not having debuginfo when -DPIC is defined.

Found another example (thanks Siddhesh).
memcpy-ssse3-back.S

Which generates:

DWARF section [13] '.debug_info' at offset 0x3964:
 [Offset]
 Compilation unit at offset 0:
 Version: 5, Abbreviation section offset: 0, Address size: 8, Offset size: 4
 Unit type: compile (1)
 [     c]  compile_unit         abbrev: 1
           stmt_list            (sec_offset) 0
           low_pc               (addr) .text.ssse3+000000000000000000 <__mempcpy_chk_ssse3_back>
           high_pc              (udata) 11014 (.text.ssse3+0x0000000000002b06)
           name                 (strp) "memcpy-ssse3-back.S"
           comp_dir             (strp) "/root/glibc/glibc-2.33.9000-984-gddcc612ce9/string"
           producer             (strp) "GNU AS 2.36.1"
           language             (data2) Mips_Assembler (32769)

[...]
Directory table:
      [path(line_strp)]
 0     ../sysdeps/x86_64/multiarch (0)
 1     ../sysdeps/x86_64/multiarch (28)

File name table:
      [path(line_strp), directory_index(udata)]
 0     memcpy-ssse3-back.S (56),  0
 1     memcpy-ssse3-back.S (76),  1

Note how the directory path zero is a relative path.
That isn't allowed, and it is unclear what it is relative to.
It is relative to the comp_dir entry.
But in DWARF5 the zero dir path entry is supposed to be equal to the comp_dir entry.

This is a bug in binutils GAS.

But we might be able to workaround it in debugedit. We do need to track the CU comp_dir and match it to the line table entry. Then if the zero dir path entry is relative (doesn't start with /) we need to prepend the comp_dir. We also need to do that for the first entry. Which is unfortunate, since normally relative paths are relative to the zero entry...

--- Additional comment from Frank Ch. Eigler on 2021-07-26 14:10:36 UTC ---

Reassigning to binutils.

Comment 5 Nick Clifton 2021-08-09 16:25:58 UTC
Fixed upstream with commit 3417bfca676.
Now waiting for ITM (and the mass rebuild to finish).

Comment 6 Nick Clifton 2021-08-16 13:46:30 UTC
Fixed in binutils-2.35.2-8.el9

Comment 9 Miloš Prchlík 2021-08-26 14:12:20 UTC
Verified with 	binutils-2.35.2-9.el9.