Bug 2182059

Summary:	eu-addr2line -i option to display inlined functions doesn't work with LTO (-flto) and Dwarf5
Product:	Red Hat Enterprise Linux 9	Reporter:	Mark Wielaard <mjw>
Component:	elfutils	Assignee:	Mark Wielaard <mjw>
elfutils sub component:	system-version	QA Contact:	Martin Cermak <mcermak>
Status:	VERIFIED ---	Docs Contact:	Petr Hybl <phybl>
Severity:	medium
Priority:	unspecified	CC:	casantos, mcermak, mjw, phybl, qe-baseos-tools-bugs, vikpatil, wcohen
Version:	9.2	Keywords:	Triaged
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	elfutils-0.189-1.el9	Doc Type:	Bug Fix
Doc Text:	This is actually 2 bugs. See also (the end of) the description. 1) -C (--demangle) plus -i (==inline) wouldn't work together Cause: eu-addr2line -C argument would made any next single character argument "disappear". Consequence: eu-addr2line -Ci would work differently from eu-addr2line -iC (the first only demangles, but does not show inlined function names) Fix: -C doesn't treat the next character as argument anymore Result: eu-addre2line -Ci will now both demangle the result and show inlined functions. 2) The eu-addr2line -i inlines option doesn't work when used on a binary compiled with gcc -flto Cause: The libdw dwarf_getscopes function would be unable to find an abstract orgin definition of a function that was compiled with gcc -flto. Consequence: eu-addr2line would be unable to show inlined functions for code compiled with the gcc -flto option. Fix: The libdw dwarf_getscopes now looks in the correct CU (compile unit) for the inlined scope. Result: eu-addr2line -i will now show inlined functions even when the binary was compiled with gcc -flto. And the libdw dwarf_getscopes function will include inlined scopes for such compiled code.	Story Points:	---
Clone Of:	2162495	Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2162495
Bug Blocks:

Description Mark Wielaard 2023-03-27 12:38:51 UTC

+++ This bug was initially created as a clone of Bug #2162495 +++

Description of problem:
eu-addr2line -i option to display inlined functions doesn't work with LTO (-flto) and Dwarf5

Version-Release number of selected component (if applicable):
elfutils-0.187-4.el8.x86_64 
gcc-toolset-11-elfutils-0.185-5.el8.x86_64

How reproducible:
every time

Steps to Reproduce:
1.From repro.sh found in repro_inlines.tgz

scl enable gcc-toolset-11 'gcc -O2 -flto -gdwarf-5 -g sample.c'

2.scl enable gcc-toolset-11 'eu-addr2line  -fCi -e a.out 0x401055'


Actual results:

"main
/mnt/data/RTVs/testIK8/elfutils/repro_inlines/sample.c:6:5"

Expected results:

# "FunctionToInline1
# /mnt/data/repro_inlines/sample.c:6
# FunctionToInline2
# /mnt/data/repro_inlines/sample.c:34
# FunctionToInline3
# /mnt/data/repro_inlines/sample.c:39
# FunctionToInline4
# /mnt/data/elfutils/repro_inlines/sample.c:43
# FunctionToInline5
# /mnt/data/repro_inlines/sample.c:47
# FunctionToInline6
# /mnt/data/repro_inlines/sample.c:51
# main
# /mnt/data/repro_inlines/sample.c:57
# "

Additional info:

--- Additional comment from Mark Wielaard on 2023-01-19 18:35:53 UTC ---

This is indeed an issue with eu-addr2line -i, also in upstream.
It doesn't seem to rely on the DWARF version with both gcc -O2 -flto -gdwarf-4 and -gdwarf-5 you don't actually get inlined subroutines.

It seems to be caused by including the -C flag:
  -C, --demangle[=ARG]       Show demangled symbols (ARG is always ignored)

Without it:

$ eu-addr2line -fi -e a.out 0x0000000000401055
FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
/tmp/repro_inlines/sample.c:6:5
FunctionToInline2
/tmp/repro_inlines/sample.c:34:5
FunctionToInline3
/tmp/repro_inlines/sample.c:39:5
FunctionToInline4
/tmp/repro_inlines/sample.c:43:5
FunctionToInline5
/tmp/repro_inlines/sample.c:47:5
FunctionToInline6
/tmp/repro_inlines/sample.c:51:5
main
/tmp/repro_inlines/sample.c:57:5

I'll investigate, it shouldn't matter whether or not we are demangling the symbols, but apparently it does.

Note that as a workaround you can probably use binutils addr2line which seems to work correctly with -fCi (at least upstream, and tested against binutils version 2.38)

--- Additional comment from Mark Wielaard on 2023-01-19 18:55:14 UTC ---

(In reply to Mark Wielaard from comment #1)
> It seems to be caused by including the -C flag:
>   -C, --demangle[=ARG]       Show demangled symbols (ARG is always ignored)
> 
> Without it:
> 
> $ eu-addr2line -fi -e a.out 0x0000000000401055
> FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
> /tmp/repro_inlines/sample.c:6:5
> FunctionToInline2
> /tmp/repro_inlines/sample.c:34:5
> FunctionToInline3
> /tmp/repro_inlines/sample.c:39:5
> FunctionToInline4
> /tmp/repro_inlines/sample.c:43:5
> FunctionToInline5
> /tmp/repro_inlines/sample.c:47:5
> FunctionToInline6
> /tmp/repro_inlines/sample.c:51:5
> main
> /tmp/repro_inlines/sample.c:57:5
> 
> I'll investigate, it shouldn't matter whether or not we are demangling the
> symbols, but apparently it does.

This is really an embarrassing bug. The problem is that -C takes an argument (which is always ignored).
The code simply treats anything as an argument, so if you use eu-readelf -fCi ... it takes that 'i' as argument to -C and ignores it.
So you don't get -i inlines...

The workaround is putting -C last (-fiC):

$ eu-addr2line -fiC -e a.out 0x0000000000401052
FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
/tmp/repro_inlines/sample.c:6:5
FunctionToInline2
/tmp/repro_inlines/sample.c:34:5
FunctionToInline3
/tmp/repro_inlines/sample.c:39:5
FunctionToInline4
/tmp/repro_inlines/sample.c:43:5
FunctionToInline5
/tmp/repro_inlines/sample.c:47:5
FunctionToInline6
/tmp/repro_inlines/sample.c:51:5
main
/tmp/repro_inlines/sample.c:57:5

--- Additional comment from Carlos Santos on 2023-01-20 13:43:57 UTC ---

The customer(In reply to Mark Wielaard from comment #2)
> (In reply to Mark Wielaard from comment #1)
> > It seems to be caused by including the -C flag:
> >   -C, --demangle[=ARG]       Show demangled symbols (ARG is always ignored)
> > 
> > Without it:
> > 
> > $ eu-addr2line -fi -e a.out 0x0000000000401055
> > FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
> > /tmp/repro_inlines/sample.c:6:5
> > FunctionToInline2
> > /tmp/repro_inlines/sample.c:34:5
> > FunctionToInline3
> > /tmp/repro_inlines/sample.c:39:5
> > FunctionToInline4
> > /tmp/repro_inlines/sample.c:43:5
> > FunctionToInline5
> > /tmp/repro_inlines/sample.c:47:5
> > FunctionToInline6
> > /tmp/repro_inlines/sample.c:51:5
> > main
> > /tmp/repro_inlines/sample.c:57:5
> > 
> > I'll investigate, it shouldn't matter whether or not we are demangling the
> > symbols, but apparently it does.
> 
> This is really an embarrassing bug. The problem is that -C takes an argument
> (which is always ignored).
> The code simply treats anything as an argument, so if you use eu-readelf
> -fCi ... it takes that 'i' as argument to -C and ignores it.
> So you don't get -i inlines...
> 
> The workaround is putting -C last (-fiC):
> 
> $ eu-addr2line -fiC -e a.out 0x0000000000401052
> FunctionToInline1 inlined at /tmp/repro_inlines/sample.c:34:5 in main
> /tmp/repro_inlines/sample.c:6:5
> FunctionToInline2
> /tmp/repro_inlines/sample.c:34:5
> FunctionToInline3
> /tmp/repro_inlines/sample.c:39:5
> FunctionToInline4
> /tmp/repro_inlines/sample.c:43:5
> FunctionToInline5
> /tmp/repro_inlines/sample.c:47:5
> FunctionToInline6
> /tmp/repro_inlines/sample.c:51:5
> main
> /tmp/repro_inlines/sample.c:57:5

The customer for whom we opened the bugs replied that the workaround does not work and I confirmed it:

$ rpm -q gcc-toolset-11
gcc-toolset-11-11.1-1.el8.x86_64

$ rpm -qf /bin/eu-addr2line
elfutils-0.187-4.el8.x86_64

$ scl enable gcc-toolset-11 'gcc -O2 -flto -gdwarf-5 -g sample.c'

$ scl enable gcc-toolset-11 'eu-addr2line  -fiC -e a.out 0x0000000000401052'
main
/work/XXXXXXXX/repro_inlines/sample.c:6:5

$ /bin/eu-addr2line  -fiC -e a.out 0x0000000000401052
main
/work/XXXXXXXX/repro_inlines/sample.c:6:5

--- Additional comment from Mark Wielaard on 2023-01-20 22:58:07 UTC ---

(In reply to Carlos Santos from comment #3)
> The customer for whom we opened the bugs replied that the workaround does
> not work and I confirmed it:
> 
> $ rpm -q gcc-toolset-11
> gcc-toolset-11-11.1-1.el8.x86_64
> 
> $ rpm -qf /bin/eu-addr2line
> elfutils-0.187-4.el8.x86_64
> 
> $ scl enable gcc-toolset-11 'gcc -O2 -flto -gdwarf-5 -g sample.c'
> 
> $ scl enable gcc-toolset-11 'eu-addr2line  -fiC -e a.out 0x0000000000401052'
> main
> /work/XXXXXXXX/repro_inlines/sample.c:6:5
> 
> $ /bin/eu-addr2line  -fiC -e a.out 0x0000000000401052

You are right.
I apparently hadn't tested with the -flto option, thinking that wasn't it. Sorry.
If you do use -flto then it really doesn't work with or without -C or placing it last.

I go and try to figure out what goes wrong when the binary is produced with -flto.
Upstream acts the same, so it isn't a RHEL specific bug.

--- Additional comment from Mark Wielaard on 2023-01-23 09:24:51 UTC ---

So there are two issues:

1) The demangle option -C prevents -i to work when used before it.
   So -Ci won't show inlines, but -iC does. Submitted an upstream patch to fix that:
   https://patchwork.sourceware.org/project/elfutils/patch/20230123091128.687801-1-mark@klomp.org/

2) The -i inlines option doesn't work when using -flto. This has now easy
   workaround. The issue is that with -flto the abstract origin of an inlined subroutine
   is in a separate "compile unit", which confused the libdw dwarf_getscopes function.
   Patch to fix that submitted upstream:
   https://patchwork.sourceware.org/project/elfutils/patch/20230123091520.688326-1-mark@klomp.org/