Bug 807053 - eu-readelf doesn't handle SHF_COMPRESSED sections
eu-readelf doesn't handle SHF_COMPRESSED sections
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: elfutils (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Roland McGrath
Fedora Extras Quality Assurance
: FutureFeature, Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-26 17:04 EDT by H.J. Lu
Modified: 2016-02-05 16:51 EST (History)
5 users (show)

See Also:
Fixed In Version: elfutils-0.165-2.fc23 elfutils-0.165-2.fc22
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-05 16:51:24 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
A hack (5.51 KB, patch)
2015-03-10 19:11 EDT, H.J. Lu
no flags Details | Diff
A patch to add elf_begin_convert (4.79 KB, application/octet-stream)
2015-03-11 16:50 EDT, H.J. Lu
no flags Details
A patch (20.22 KB, patch)
2015-03-11 19:01 EDT, H.J. Lu
no flags Details | Diff
A cleaned up patch (20.22 KB, application/octet-stream)
2015-03-11 19:44 EDT, H.J. Lu
no flags Details
An updated patch to use INTUSE (20.35 KB, patch)
2015-03-12 10:57 EDT, H.J. Lu
no flags Details | Diff
A patch to add a testcase (22.36 KB, patch)
2015-03-12 11:49 EDT, H.J. Lu
no flags Details | Diff

  None (edit)
Description H.J. Lu 2012-03-26 17:04:37 EDT
[hjl@gnu-6 src]$ cat x.c
int
foo (void)
{
  return 1;
}
[hjl@gnu-6 src]$ gcc -S -g x.c
[hjl@gnu-6 src]$ as --compress-debug-sections -o x.o x.s
[hjl@gnu-6 src]$ ./readelf -w x.o
./readelf: cannot get debug context descriptor: No DWARF information found

DWARF section [ 4] '.zdebug_info' at offset 0x4c:
 [Offset]

DWARF section [ 6] '.zdebug_abbrev' at offset 0x88:
 [ Code]
Segmentation fault
[hjl@gnu-6 src]$
Comment 1 Mark Wielaard 2012-03-26 17:41:28 EDT
Confirmed against $ eu-readelf --version
readelf (Red Hat elfutils) 0.153
$ rpm -q elfutils
elfutils-0.153-1.fc16.x86_64

(gdb) run
Starting program: /usr/bin/eu-readelf -w x.o
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
/usr/bin/eu-readelf: cannot get debug context descriptor: No DWARF information found

DWARF section [ 4] '.zdebug_info' at offset 0x4c:
 [Offset]

DWARF section [ 6] '.zdebug_abbrev' at offset 0x86:
 [ Code]

Program received signal SIGSEGV, Segmentation fault.
0x000000000040af60 in print_debug_abbrev_section (dwflmod=<optimized out>, 
    ebl=<optimized out>, ehdr=<optimized out>, scn=<optimized out>, 
    shdr=<optimized out>, dbg=0x7fffffffd9c0) at readelf.c:4612
4612	  while (offset < dbg->sectiondata[IDX_debug_abbrev]->d_size)
Comment 2 Mark Wielaard 2012-03-27 10:11:58 EDT
This is weird. A hand build libdw seems fine. But at least with the libdw from elfutils-0.153-1.fc16.x86_64 zrc = inflate (&z, Z_FINISH); libdw/dwarf_begin_elf.c:199 returns Z_DATA_ERROR (-3) for the .zdebug_info section (the others seem to decode fine).
Comment 3 Mark Wielaard 2012-03-27 11:53:53 EDT
My current guess is that the following in libdw/dwarf_begin_elf.c (check_section) isn't enough:

            z_stream z =
              {
                .next_in = data->d_buf + 4 + 8,
                .avail_in = data->d_size - 4 - 8,
                .next_out = zdata->d_buf,
                .avail_out = zdata->d_size
              };

And that we also have to initialize (to zero) the following fields:

    alloc_func zalloc;  /* used to allocate the internal state */
    free_func  zfree;   /* used to free the internal state */
    voidpf     opaque;  /* private data object passed to zalloc and zfree */

Which might explain why the results seemed a bit random (since z is stack allocated and might be initialized with random bits).
Comment 4 Mark Wielaard 2012-03-27 15:28:37 EDT
On irc Frank made the observation that the data in the section seemed to be "corrupted", and suggested that might be because libdwfl relocates the section (it is from an ET_REL file) before decompression. That seems indeed to be the issue. Disabling relocate_section in libdwfl/relocate.c makes the decompression go fine.
Comment 5 Roland McGrath 2012-03-27 15:39:34 EDT
The comment#3 theory doesn't make sense.  That initializer syntax implicitly sets all unnamed fields to zero.

The real reason in comment#4 does make sense.  The way we've implemented the compressed-section support just doesn't jibe with the "pure" libdwfl/libdw layering of the original relocation approach.  We'd need to violate that layering and make libdwfl do the decompression before it does relocation, so libdw sees only the uncompressed, relocated data.  OTOH, this should be a non-issue in the new roland/relocate branch, so perhaps this is motivation to finally tie up the loose ends there and merge in that work.

Of course, this all assumes that on the producer side the relocatable case is being handled as we expect.  i.e., that e.g. a .rela.zdebug_info section uses offsets consistent with the uncompressed .debug_info section image that got turned into .zdebug_info.  That's the only sensical thing to do, but it's also rather a strange format.  Really it seems quite questionable that anybody emitted compressed-sections format in an ET_REL file to begin with.  Who does that?
Comment 6 Mark Wielaard 2012-03-27 15:55:20 EDT
I don't think we can easily support .z compressed sections for ET_REL files till we finish the roland/relocate branch for lazy relocations.

Maybe we could special case .z compressed sections in __libdwfl_relocate(). But then we would need some new interface to do the relocations on the uncompressed data section later.
Comment 7 Fedora End Of Life 2012-08-07 14:14:57 EDT
This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 8 Mark Wielaard 2012-08-07 20:04:29 EDT
We still don't handle this.
Comment 9 Fedora End Of Life 2013-04-03 15:43:32 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 10 H.J. Lu 2015-03-10 11:43:45 EDT
It was fixed in elfutils git:

commit f7cdc74ed34adc539186385342d99b357c3dbb6b
Author: Mark Wielaard <mjw@redhat.com>
Date:   Wed Apr 9 22:38:07 2014 +0200

    readelf: Read the decompressed data when processing .zdebug DWARF sections.
    
    readelf uses libdw to open the Dwarf and read some of the DWARF data.
    But it also uses its own parsers to display some of the low-level
    unprocessed data. If the DWARF debug section was zlib compressed it
    should actually use the decompressed section data from libdw instead
    of the raw section data.
    
    Includes a testcase for those sections that couldn't be properly
    displayed when compressed before.
    
    Signed-off-by: Mark Wielaard <mjw@redhat.com>

Git tag elfutils-0.161 works correctly.  But elfutils-0.161-2.fc20.x86_64
doesn't.
Comment 11 H.J. Lu 2015-03-10 12:27:37 EDT
The problem is /usr/lib64/elfutils:

[root@gnu-6 src]# eu-readelf -w -a readelf.o > /dev/null
eu-readelf: cannot get debug context descriptor: No DWARF information found
eu-readelf: cannot get .debug_loc content: no error
eu-readelf: cannot get .debug_aranges content: no error
eu-readelf: cannot get .debug_ranges content: no error
eu-readelf: cannot get line data section data: no error
[root@gnu-6 src]# mv /usr/lib64/elfutils /usr/lib64/elfutils.saved
[root@gnu-6 src]# eu-readelf -w -a readelf.o > /dev/null
[root@gnu-6 src]#
Comment 12 Mark Wielaard 2015-03-10 17:03:17 EDT
(In reply to H.J. Lu from comment #10)
> It was fixed in elfutils git:
> 
> commit f7cdc74ed34adc539186385342d99b357c3dbb6b
> Author: Mark Wielaard <mjw@redhat.com>
> Date:   Wed Apr 9 22:38:07 2014 +0200
> 
>     readelf: Read the decompressed data when processing .zdebug DWARF
> sections.
>     
>     readelf uses libdw to open the Dwarf and read some of the DWARF data.
>     But it also uses its own parsers to display some of the low-level
>     unprocessed data. If the DWARF debug section was zlib compressed it
>     should actually use the decompressed section data from libdw instead
>     of the raw section data.
>     
>     Includes a testcase for those sections that couldn't be properly
>     displayed when compressed before.
>     
>     Signed-off-by: Mark Wielaard <mjw@redhat.com>

That would "fix" things for non-ET_REL files. Given there are no relocations against the .[z]debug sections. The real problem is that .zdebug sections are ill-defined when there are relocations against the compressed sections. The current setup of the code (relocate.c in libdwfl) will apply the relocations against the section data, then the section will be decompressed when used as DWARF debug data (dwarf_begin_elf.c in libdw).
 
> Git tag elfutils-0.161 works correctly.  But elfutils-0.161-2.fc20.x86_64
> doesn't.

How are you testing?
Comment 13 Mark Wielaard 2015-03-10 17:07:12 EDT
(In reply to H.J. Lu from comment #11)
> The problem is /usr/lib64/elfutils:
> 
> [root@gnu-6 src]# eu-readelf -w -a readelf.o > /dev/null
> eu-readelf: cannot get debug context descriptor: No DWARF information found
> eu-readelf: cannot get .debug_loc content: no error
> eu-readelf: cannot get .debug_aranges content: no error
> eu-readelf: cannot get .debug_ranges content: no error
> eu-readelf: cannot get line data section data: no error
> [root@gnu-6 src]# mv /usr/lib64/elfutils /usr/lib64/elfutils.saved
> [root@gnu-6 src]# eu-readelf -w -a readelf.o > /dev/null

I am surprised eu-readelf still works when removing the ebl backend code. It probably should explictly fail instead of limping along. When you do that there will be errors about anything that is machine specific in the ELF file or DWARF data since ebl openbackend will just fail. You'll see lots of INVALID data in the eu-readelf output.
Comment 14 H.J. Lu 2015-03-10 19:11:41 EDT
Created attachment 1000196 [details]
A hack

This a hack to make it work for readelf.  But it doesn't work with

strip  --reloc-debug-sections

Ideally elf_getdata should automatically decompress the debug sections.
Maybe we can add a callback to convert_data to decompress the debug
sections.
Comment 15 H.J. Lu 2015-03-11 16:50:46 EDT
Created attachment 1000681 [details]
A patch to add elf_begin_convert

This still doesn't work with strip --reloc-debug-sections.
I will try to fix it.
Comment 16 H.J. Lu 2015-03-11 19:01:10 EDT
Created attachment 1000712 [details]
A patch

This patch passed "make check" on x86-64 and x86-32 with assembler from
binutils 2.26, which generates compressed debug section by default.
Comment 17 H.J. Lu 2015-03-11 19:44:22 EDT
Created attachment 1000728 [details]
A cleaned up patch
Comment 18 Mark Wielaard 2015-03-12 06:52:10 EDT
It would be best if you could first discuss your design on the upstream mailinglist:
elfutils-devel@lists.fedorahosted.org.
https://fedorahosted.org/mailman/listinfo/elfutils-devel
https://git.fedorahosted.org/cgit/elfutils.git/plain/CONTRIBUTING

Which make check tests fail before your patch? Adding some explicit test cases would be good, so they can be tested also on platforms that don't use .zdebug sections in ET_REL files.

Having to change all programs that depend on libelf now having to link against libdw and use a special case new entry point (dwarf_elf_begin) isn't very attracive. And what about programs that already use libdw dwarf_begin_elf? Those would need to be rewritten?

Exporting a new libelf function should be discussed first to see how other libelf implementations handle this.
Comment 19 H.J. Lu 2015-03-12 10:57:30 EDT
Created attachment 1001009 [details]
An updated patch to use INTUSE
Comment 20 H.J. Lu 2015-03-12 11:03:08 EDT
(In reply to Mark Wielaard from comment #18)
> It would be best if you could first discuss your design on the upstream
> mailinglist:
> elfutils-devel@lists.fedorahosted.org.
> https://fedorahosted.org/mailman/listinfo/elfutils-devel
> https://git.fedorahosted.org/cgit/elfutils.git/plain/CONTRIBUTING

I will send emails to elfutils-devel.

> Which make check tests fail before your patch? Adding some explicit test
> cases would be good, so they can be tested also on platforms that don't use
> .zdebug sections in ET_REL files.

I will add some tests.

> Having to change all programs that depend on libelf now having to link
> against libdw and use a special case new entry point (dwarf_elf_begin) isn't

I am open to all suggestions.  I thought about updating elf_begin.
But it seems to defeat the purpose of libdw. If a program wants to
deal with DWARF, it should use dwarf_elf_begin.

> very attracive. And what about programs that already use libdw
> dwarf_begin_elf? Those would need to be rewritten?

Did you mean dwarf_begin?  My change updated dwarf_begin to call
dwarf_elf_begin instead of elf_begin.

> Exporting a new libelf function should be discussed first to see how other
> libelf implementations handle this.

I doubt other libelfs support compressed debug sections.
Comment 21 H.J. Lu 2015-03-12 11:49:48 EDT
Created attachment 1001037 [details]
A patch to add a testcase
Comment 22 H.J. Lu 2015-03-12 12:05:24 EDT
(In reply to Mark Wielaard from comment #18)
> It would be best if you could first discuss your design on the upstream
> mailinglist:
> elfutils-devel@lists.fedorahosted.org.

I have sent an email, which is waiting for moderator approval.
Comment 23 Mark Wielaard 2015-03-12 12:07:54 EDT
(In reply to H.J. Lu from comment #20)
> (In reply to Mark Wielaard from comment #18)
> > And what about programs that already use libdw
> > dwarf_begin_elf? Those would need to be rewritten?
> 
> Did you mean dwarf_begin?  My change updated dwarf_begin to call
> dwarf_elf_begin instead of elf_begin.

There are two ways to get a Dwarf handle from libdw. dwarf_begin, which takes a file descriptor. And dwarf_begin_elf, which takes an Elf handle.

> > Exporting a new libelf function should be discussed first to see how other
> > libelf implementations handle this.
> 
> I doubt other libelfs support compressed debug sections.

Probably not. But SHF_COMPRESSED did get standardized and at least Solaris seems to support it. They might have also altered their libelf implementation to deal with it: https://groups.google.com/forum/#!topic/generic-abi/dBOS1H47Q64
Comment 24 H.J. Lu 2015-03-12 12:19:58 EDT
(In reply to Mark Wielaard from comment #23)
> (In reply to H.J. Lu from comment #20)
> > (In reply to Mark Wielaard from comment #18)
> > > And what about programs that already use libdw
> > > dwarf_begin_elf? Those would need to be rewritten?
> > 
> > Did you mean dwarf_begin?  My change updated dwarf_begin to call
> > dwarf_elf_begin instead of elf_begin.
> 
> There are two ways to get a Dwarf handle from libdw. dwarf_begin, which
> takes a file descriptor. And dwarf_begin_elf, which takes an Elf handle.

The Elf handle should be created with dwarf_elf_begin if the user
wants to support DWARF debug sections.

> > > Exporting a new libelf function should be discussed first to see how other
> > > libelf implementations handle this.
> > 
> > I doubt other libelfs support compressed debug sections.
> 
> Probably not. But SHF_COMPRESSED did get standardized and at least Solaris
> seems to support it. They might have also altered their libelf
> implementation to deal with it:
> https://groups.google.com/forum/#!topic/generic-abi/dBOS1H47Q64

SHF_COMPRESSED doesn't help the existing relocatable files with compressed
debug sections.  But infrastructure update in libelf for SHF_COMPRESSED
can be used to support compressed debug sections.
Comment 25 H.J. Lu 2015-03-12 12:44:27 EDT
NB.  strip --reloc-debug-sections will decompress the compressed
debug sections and resolve all trivial relocations.  It will generate
the decompressed debug sections with the compressed debug section
names.  Both readelf and eu-readelf can handle them correctly.
Comment 26 Mark Wielaard 2015-03-13 08:47:07 EDT
Moved discussion upstream:
https://lists.fedorahosted.org/pipermail/elfutils-devel/2015-March/004623.html
Comment 27 H.J. Lu 2015-04-15 08:32:04 EDT
(In reply to Mark Wielaard from comment #23)
> 
> Probably not. But SHF_COMPRESSED did get standardized and at least Solaris
> seems to support it. They might have also altered their libelf
> implementation to deal with it:
> https://groups.google.com/forum/#!topic/generic-abi/dBOS1H47Q64

SHF_COMPRESSED support has been added to gas/binutils/ld on binutils
master branch. elfutils should handle compressed debug sections
properly.
Comment 28 Mark Wielaard 2015-04-15 08:42:37 EDT
(In reply to H.J. Lu from comment #27)
> (In reply to Mark Wielaard from comment #23)
> > 
> > Probably not. But SHF_COMPRESSED did get standardized and at least Solaris
> > seems to support it. They might have also altered their libelf
> > implementation to deal with it:
> > https://groups.google.com/forum/#!topic/generic-abi/dBOS1H47Q64
> 
> SHF_COMPRESSED support has been added to gas/binutils/ld on binutils
> master branch. elfutils should handle compressed debug sections
> properly.

So how does the solaris libelf implementation handle SHF_COMPRESSED sections?
We should probably match any interface changes they made. In particular how do elf_getdata () and elf_newdata () work for SHF_COMPRESSED sections? Does the compression happen transparently (and does elf_rawdata provide the compressed data) or is this completely up to the user (which would be a pain and wouldn't really help with better compressed section support since it would mean to audit all elf section data accesses in all tools and other libraries based on libelf).
Comment 29 H.J. Lu 2015-04-15 08:49:11 EDT
(In reply to Mark Wielaard from comment #28)
> So how does the solaris libelf implementation handle SHF_COMPRESSED sections?
> We should probably match any interface changes they made. In particular how
> do elf_getdata () and elf_newdata () work for SHF_COMPRESSED sections? Does
> the compression happen transparently (and does elf_rawdata provide the
> compressed data) or is this completely up to the user (which would be a pain
> and wouldn't really help with better compressed section support since it
> would mean to audit all elf section data accesses in all tools and other
> libraries based on libelf).

I think the existing functions should decompress compressed section
transparently.  A new function should be added to return compressed
content.  We should check with Oracle.
Comment 30 H.J. Lu 2015-04-15 16:53:56 EDT
(In reply to Mark Wielaard from comment #28)
> So how does the solaris libelf implementation handle SHF_COMPRESSED sections?
> We should probably match any interface changes they made. In particular how
> do elf_getdata () and elf_newdata () work for SHF_COMPRESSED sections? Does
> the compression happen transparently (and does elf_rawdata provide the
> compressed data) or is this completely up to the user (which would be a pain
> and wouldn't really help with better compressed section support since it
> would mean to audit all elf section data accesses in all tools and other
> libraries based on libelf).

See

https://groups.google.com/forum/#!topic/generic-abi/9ZgNqIC2X-4
Comment 31 Mark Wielaard 2015-06-08 08:24:49 EDT
This really should not have been reopened and retitiled. But should have been a separate bug from the original one (ET_REL files with compressed debug sections not handled). Lets retitle again to make more clear what we are discussing now.
Comment 32 Mark Wielaard 2016-01-07 07:39:06 EST
This has been fixed as part of the compressed ELF section support patch series to be included in elfutils 0.165 which will be released in a couple of days.
https://lists.fedorahosted.org/archives/list/elfutils-devel%40lists.fedorahosted.org/thread/CJBIABJUMXBAFBLB6MM5C3SXRYDXZ2MP/
Comment 33 Fedora Update System 2016-01-18 09:39:37 EST
elfutils-0.165-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-a6cf456e31
Comment 34 Fedora Update System 2016-01-19 22:55:08 EST
elfutils-0.165-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-a6cf456e31
Comment 35 Fedora Update System 2016-01-20 16:51:33 EST
elfutils-0.165-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 36 Fedora Update System 2016-01-25 06:30:55 EST
elfutils-0.165-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-35e3877a89
Comment 37 Fedora Update System 2016-01-25 23:23:16 EST
elfutils-0.165-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-35e3877a89
Comment 38 Fedora Update System 2016-02-05 16:51:13 EST
elfutils-0.165-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.