Bug 2182059 - eu-addr2line -i option to display inlined functions doesn't work with LTO (-flto) and Dwarf5
Summary: eu-addr2line -i option to display inlined functions doesn't work with LTO (-f...
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: elfutils
Version: 9.2
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Mark Wielaard
QA Contact: Martin Cermak
Petr Hybl
URL:
Whiteboard:
Depends On: 2162495
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-27 12:38 UTC by Mark Wielaard
Modified: 2023-07-19 11:51 UTC (History)
7 users (show)

Fixed In Version: elfutils-0.189-1.el9
Doc Type: Bug Fix
Doc Text:
This is actually 2 bugs. See also (the end of) the description. 1) -C (--demangle) plus -i (==inline) wouldn't work together Cause: eu-addr2line -C argument would made any next single character argument "disappear". Consequence: eu-addr2line -Ci would work differently from eu-addr2line -iC (the first only demangles, but does not show inlined function names) Fix: -C doesn't treat the next character as argument anymore Result: eu-addre2line -Ci will now both demangle the result and show inlined functions. 2) The eu-addr2line -i inlines option doesn't work when used on a binary compiled with gcc -flto Cause: The libdw dwarf_getscopes function would be unable to find an abstract orgin definition of a function that was compiled with gcc -flto. Consequence: eu-addr2line would be unable to show inlined functions for code compiled with the gcc -flto option. Fix: The libdw dwarf_getscopes now looks in the correct CU (compile unit) for the inlined scope. Result: eu-addr2line -i will now show inlined functions even when the binary was compiled with gcc -flto. And the libdw dwarf_getscopes function will include inlined scopes for such compiled code.
Clone Of: 2162495
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-153165 0 None None None 2023-03-27 12:40:47 UTC

Description Mark Wielaard 2023-03-27 12:38:51 UTC 
+++ This bug was initially created as a clone of Bug #2162495 +++

Description of problem:
eu-addr2line -i option to display inlined functions doesn't work with LTO (-flto) and Dwarf5

Version-Release number of selected component (if applicable):
elfutils-0.187-4.el8.x86_64 
gcc-toolset-11-elfutils-0.185-5.el8.x86_64

How reproducible:
every time

Steps to Reproduce:
1.From repro.sh found in repro_inlines.tgz

scl enable gcc-toolset-11 'gcc -O2 -flto -gdwarf-5 -g sample.c'

2.scl enable gcc-toolset-11 'eu-addr2line  -fCi -e a.out 0x401055'


Actual results:

"main
/mnt/data/RTVs/testIK8/elfutils/repro_inlines/sample.c:6:5"

Expected results:

# "FunctionToInline1
# /mnt/data/repro_inlines/sample.c:6
# FunctionToInline2
# /mnt/data/repro_inlines/sample.c:34
# FunctionToInline3
# /mnt/data/repro_inlines/sample.c:39
# FunctionToInline4
# /mnt/data/elfutils/repro_inlines/sample.c:43
# FunctionToInline5
# /mnt/data/repro_inlines/sample.c:47
# FunctionToInline6
# /mnt/data/repro_inlines/sample.c:51
# main
# /mnt/data/repro_inlines/sample.c:57
# "

Additional info:

--- Additional comment from Mark Wielaard on 2023-01-19 18:35:53 UTC ---

This is indeed an issue with eu-addr2line -i, also in upstream.
It doesn't seem to rely on the DWARF version with both gcc -O2 -flto -gdwarf-4 and -gdwarf-5 you don't actually get inlined subroutines.

It seems to be caused by including the -C flag:
  -C, --demangle[=ARG]       Show demangled symbols (ARG is always ignored)

Without it:

$ eu-addr2line -fi -e a.out 0x0000000000401055
FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
/tmp/repro_inlines/sample.c:6:5
FunctionToInline2
/tmp/repro_inlines/sample.c:34:5
FunctionToInline3
/tmp/repro_inlines/sample.c:39:5
FunctionToInline4
/tmp/repro_inlines/sample.c:43:5
FunctionToInline5
/tmp/repro_inlines/sample.c:47:5
FunctionToInline6
/tmp/repro_inlines/sample.c:51:5
main
/tmp/repro_inlines/sample.c:57:5

I'll investigate, it shouldn't matter whether or not we are demangling the symbols, but apparently it does.

Note that as a workaround you can probably use binutils addr2line which seems to work correctly with -fCi (at least upstream, and tested against binutils version 2.38)

--- Additional comment from Mark Wielaard on 2023-01-19 18:55:14 UTC ---

(In reply to Mark Wielaard from comment #1)
> It seems to be caused by including the -C flag:
>   -C, --demangle[=ARG]       Show demangled symbols (ARG is always ignored)
> 
> Without it:
> 
> $ eu-addr2line -fi -e a.out 0x0000000000401055
> FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
> /tmp/repro_inlines/sample.c:6:5
> FunctionToInline2
> /tmp/repro_inlines/sample.c:34:5
> FunctionToInline3
> /tmp/repro_inlines/sample.c:39:5
> FunctionToInline4
> /tmp/repro_inlines/sample.c:43:5
> FunctionToInline5
> /tmp/repro_inlines/sample.c:47:5
> FunctionToInline6
> /tmp/repro_inlines/sample.c:51:5
> main
> /tmp/repro_inlines/sample.c:57:5
> 
> I'll investigate, it shouldn't matter whether or not we are demangling the
> symbols, but apparently it does.

This is really an embarrassing bug. The problem is that -C takes an argument (which is always ignored).
The code simply treats anything as an argument, so if you use eu-readelf -fCi ... it takes that 'i' as argument to -C and ignores it.
So you don't get -i inlines...

The workaround is putting -C last (-fiC):

$ eu-addr2line -fiC -e a.out 0x0000000000401052
FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
/tmp/repro_inlines/sample.c:6:5
FunctionToInline2
/tmp/repro_inlines/sample.c:34:5
FunctionToInline3
/tmp/repro_inlines/sample.c:39:5
FunctionToInline4
/tmp/repro_inlines/sample.c:43:5
FunctionToInline5
/tmp/repro_inlines/sample.c:47:5
FunctionToInline6
/tmp/repro_inlines/sample.c:51:5
main
/tmp/repro_inlines/sample.c:57:5

--- Additional comment from Carlos Santos on 2023-01-20 13:43:57 UTC ---

The customer(In reply to Mark Wielaard from comment #2)
> (In reply to Mark Wielaard from comment #1)
> > It seems to be caused by including the -C flag:
> >   -C, --demangle[=ARG]       Show demangled symbols (ARG is always ignored)
> > 
> > Without it:
> > 
> > $ eu-addr2line -fi -e a.out 0x0000000000401055
> > FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
> > /tmp/repro_inlines/sample.c:6:5
> > FunctionToInline2
> > /tmp/repro_inlines/sample.c:34:5
> > FunctionToInline3
> > /tmp/repro_inlines/sample.c:39:5
> > FunctionToInline4
> > /tmp/repro_inlines/sample.c:43:5
> > FunctionToInline5
> > /tmp/repro_inlines/sample.c:47:5
> > FunctionToInline6
> > /tmp/repro_inlines/sample.c:51:5
> > main
> > /tmp/repro_inlines/sample.c:57:5
> > 
> > I'll investigate, it shouldn't matter whether or not we are demangling the
> > symbols, but apparently it does.
> 
> This is really an embarrassing bug. The problem is that -C takes an argument
> (which is always ignored).
> The code simply treats anything as an argument, so if you use eu-readelf
> -fCi ... it takes that 'i' as argument to -C and ignores it.
> So you don't get -i inlines...
> 
> The workaround is putting -C last (-fiC):
> 
> $ eu-addr2line -fiC -e a.out 0x0000000000401052
> FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
> /tmp/repro_inlines/sample.c:6:5
> FunctionToInline2
> /tmp/repro_inlines/sample.c:34:5
> FunctionToInline3
> /tmp/repro_inlines/sample.c:39:5
> FunctionToInline4
> /tmp/repro_inlines/sample.c:43:5
> FunctionToInline5
> /tmp/repro_inlines/sample.c:47:5
> FunctionToInline6
> /tmp/repro_inlines/sample.c:51:5
> main
> /tmp/repro_inlines/sample.c:57:5

The customer for whom we opened the bugs replied that the workaround does not work and I confirmed it:

$ rpm -q gcc-toolset-11
gcc-toolset-11-11.1-1.el8.x86_64

$ rpm -qf /bin/eu-addr2line
elfutils-0.187-4.el8.x86_64

$ scl enable gcc-toolset-11 'gcc -O2 -flto -gdwarf-5 -g sample.c'

$ scl enable gcc-toolset-11 'eu-addr2line  -fiC -e a.out 0x0000000000401052'
main
/work/XXXXXXXX/repro_inlines/sample.c:6:5

$ /bin/eu-addr2line  -fiC -e a.out 0x0000000000401052
main
/work/XXXXXXXX/repro_inlines/sample.c:6:5

--- Additional comment from Mark Wielaard on 2023-01-20 22:58:07 UTC ---

(In reply to Carlos Santos from comment #3)
> The customer for whom we opened the bugs replied that the workaround does
> not work and I confirmed it:
> 
> $ rpm -q gcc-toolset-11
> gcc-toolset-11-11.1-1.el8.x86_64
> 
> $ rpm -qf /bin/eu-addr2line
> elfutils-0.187-4.el8.x86_64
> 
> $ scl enable gcc-toolset-11 'gcc -O2 -flto -gdwarf-5 -g sample.c'
> 
> $ scl enable gcc-toolset-11 'eu-addr2line  -fiC -e a.out 0x0000000000401052'
> main
> /work/XXXXXXXX/repro_inlines/sample.c:6:5
> 
> $ /bin/eu-addr2line  -fiC -e a.out 0x0000000000401052

You are right.
I apparently hadn't tested with the -flto option, thinking that wasn't it. Sorry.
If you do use -flto then it really doesn't work with or without -C or placing it last.

I go and try to figure out what goes wrong when the binary is produced with -flto.
Upstream acts the same, so it isn't a RHEL specific bug.

--- Additional comment from Mark Wielaard on 2023-01-23 09:24:51 UTC ---

So there are two issues:

1) The demangle option -C prevents -i to work when used before it.
   So -Ci won't show inlines, but -iC does. Submitted an upstream patch to fix that:
   https://patchwork.sourceware.org/project/elfutils/patch/20230123091128.687801-1-mark@klomp.org/

2) The -i inlines option doesn't work when using -flto. This has now easy
   workaround. The issue is that with -flto the abstract origin of an inlined subroutine
   is in a separate "compile unit", which confused the libdw dwarf_getscopes function.
   Patch to fix that submitted upstream:
   https://patchwork.sourceware.org/project/elfutils/patch/20230123091520.688326-1-mark@klomp.org/
Note You need to log in before you can comment on or make changes to this bug.