Bug 1311352 - objdump -S disassembly code doesn't follow /usr/lib/debug/.... conventions, so can't find sources
objdump -S disassembly code doesn't follow /usr/lib/debug/.... conventions,...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: binutils (Show other bugs)
7.2
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Nick Clifton
Miloš Prchlík
: Patch
Depends On:
Blocks: 1311494 1311792
  Show dependency treegraph
 
Reported: 2016-02-23 20:16 EST by Frank Ch. Eigler
Modified: 2016-11-03 21:55 EDT (History)
10 users (show)

See Also:
Fixed In Version: binutils-2.25.1-10.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1311494 (view as bug list)
Environment:
Last Closed: 2016-11-03 21:55:11 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Add Red Hat specific search locations for debug info files (1.29 KB, patch)
2016-02-24 05:38 EST, Nick Clifton
no flags Details | Diff
Prevent attempts to read an corrupt DWARF DIE value (1.07 KB, patch)
2016-02-26 05:48 EST, Nick Clifton
no flags Details | Diff

  None (edit)
Description Frank Ch. Eigler 2016-02-23 20:16:55 EST
binutils-2.23.52.0.1-55.el7.x86_64
nscd-2.17-106.el7_2.1.x86_64
glibc-debuginfo-2.17-106.el7_2.1.x86_64

strace -eopen objdump -drS /usr/bin/nscd

... shows references to 

/usr/sbin/nscd.debug
/usr/sbin/.debug/nscd.debug
/usr/lib64/debug/usr/sbin/nscd.debug

but not actually

/usr/lib/debug/lib64/nscd.debug
nor /usr/lib/debug/.build_id/95/b5.....etc.

so objdump -S can't show us sources.
Comment 1 Frank Ch. Eigler 2016-02-23 20:18:32 EST
(sorry, s,lib64,usr/sbin, in the 'not actually' list above; was mixing two separate experiments)
Comment 3 Frank Ch. Eigler 2016-02-23 20:29:50 EST
(same for addr2line)

The .gnu_debuglink section for this binary says:

Hex dump of section '.gnu_debuglink':
  0x00000000 6e736364 2e6346562 75670000 54adf413   nscd.debug..T...
Comment 4 Nick Clifton 2016-02-24 04:41:58 EST
Hi Frank,

  Are the conventions for locating debug files documented somewhere ?  If so, please could you point me at them ?

Cheers
  Nick

PS.  Where can I find the nscd-debug rpm ?  It does not appear to be part of vanilla F23...
Comment 5 Brad Hubbard 2016-02-24 05:01:40 EST
Hi Nick, The debuginfo for nscd is in the glibc-debuginfo-common package as nscd is a sub-package of glibc. This should get installed if you install the glibc-debuginfo package. If fedora is affected however, this bug should be cloned since this bug is for the rhel7 release.

$ dnf provides \*/nscd.debug
Repository InstallMedia has no mirror or baseurl set.                                 
Last metadata expiration check performed 0:01:10 ago on Wed Feb 24 19:56:17 2016.
glibc-debuginfo-common-2.20-7.fc21.x86_64 : Debug information for package glibc
Repo        : @System

I'm afraid I'm not aware of the documentation for debuginfo paths so I'll let Frank answer that one.
Comment 6 Brad Hubbard 2016-02-24 05:05:16 EST
I should also note that nscd is not the only package affected. This behaviour was initially noticed when working with the ceph-mon binary from Red Hat Ceph Storage. I then tried nscd and found it also was affected so I think there's a good chance many, if not all, rhel7 binaries will demonstrate this behaviour. Not to suggest this is necessarily a problem with the binaries themselves as that is yet to be determined positively.
Comment 7 Nick Clifton 2016-02-24 05:38 EST
Created attachment 1130176 [details]
Add Red Hat specific search locations for debug info files

Hi Guys,

  Ah - the bug does not exist in Fedora, which explains why I was confused.
  Fortunately it also leads to an answer - the Fedora binutils include a 
  patch specifically to add extra search paths to the find_separate_debug_file
  function.

  The uploaded patch adds this functionality to RHEL 7.2.  I cannot apply it
  until I receive all the required ACKs however, but feel free to test it out
  locally.

  The same problem exists in RHEL 7.3. so if this BZ can be cloned for that
  release then I can apply the same patch there...

Cheers
  Nick
Comment 8 Brad Hubbard 2016-02-24 06:15:41 EST
Done Nick,

bz 1311494

I'll test the patch tomorrow as it's getting late for me. 

If it's not too much trouble could you explain how this bug came about as I'm sure this used to work on rhel6, if not on 7.1.

Many thanks for your efforts.
Comment 9 Nick Clifton 2016-02-24 07:15:34 EST
Hi Brad,

> bz 1311494

Thanks.

> If it's not too much trouble could you explain how this bug came about as
> I'm sure this used to work on rhel6, if not on 7.1.

Really ?  Then I am not sure.  Both 7.1 and 6.8 binutils have support for 
separate debuginfo files, but neither of them have the patch to search 
below /usr/lib/debug, so in theory they ought to be exhibiting the same 
problem.

Maybe the installation of debuginfo files into /usr/lib/debug is a more recent
change ?  64-bit RHEL6 searches below /usr/lib64/debug so maybe that is where
the debuginfo files are installed ?

Cheers
  Nick
Comment 10 Brad Hubbard 2016-02-24 21:54:59 EST
I worked out why I thought this used to work.

I found the old document where I'd written about the use of "objdump -rdS" and realised that I had been running it on a binary I had compiled from source that was unstripped.

Now, on to testing the patch.
Comment 11 Brad Hubbard 2016-02-25 02:34:11 EST
Hi Nick,

I built a test package including your patch and it seems to work fine for the nscd binary but when I try it on ceph-mon I get the following.

# objdump -rdS /usr/bin/ceph-mon &> output
Segmentation fault

# head output

/usr/bin/ceph-mon:     file format elf64-x86-64


Disassembly of section .init:

00000000005484f0 <_init>:
BFD: Dwarf Error: Offset (4207821855) greater than or equal to .debug_str size (39495439).
BFD: Dwarf Error: Offset (4245841173) greater than or equal to .debug_str size (39495439).
BFD: Dwarf Error: Offset (3958553601) greater than or equal to .debug_str size (39495439).

# tail output
  }

  bool should_gather(unsigned sub, int level) {
  54afcd:       50                      push   %rax
    assert(sub < m_subsys.size());
  54afce:       48 8d 35 9b 99 3b 00    lea    0x3b999b(%rip),%rsi        # 904970 <_IO_stdin_used+0x10>
  54afd5:       48 8d 3d a9 99 3b 00    lea    0x3b99a9(%rip),%rdi        # 904985 <_IO_stdin_used+0x25>
  54afdc:       ba 3e 00 00 00          mov    $0x3e,%edx
  54afe1:       e8 5a af 26 00          callq  7b5f40 <_ZN4ceph18__ceph_assert_failEPKcS1_iS1_>
BFD: Dwarf Error: Offset (1074004614) greater than or equal to .debug_str size (39495439).

So it's kind of working as I can see some source output in the output but all the dwarf errors and the segfault are, of course, a show stopper.
Comment 12 Nick Clifton 2016-02-25 08:28:33 EST
Hi Brad,

> I built a test package including your patch and it seems to work fine for
> the nscd binary but when I try it on ceph-mon I get the following.
> 
> # objdump -rdS /usr/bin/ceph-mon &> output
> Segmentation fault

Darn.  Can you point me at a machine where I can try this out for myself please ?  (I only have 5Gb left on my root partition and that is not enough to install ceph and its debuginfo.  :-(

Cheers
  Nick
Comment 14 Nick Clifton 2016-02-26 05:16:56 EST
Hi Brad,

  Hmm, tricky.  I can now reproduce the problem.  Essentially what is happening
is that corrupt DWARF information is causing the BFD library to attempt to read
a LEB128 encoded value at an illegal address.  The problem is, how to fix this.

  A while ago I spent some time hardening the BFD library so that it would
catch problems like this and report the corruption, rather than performing
illegal reads.  The problem is that this was a very extensive patch, and
backporting it in to RHEL-7.2 would take a long time, and possibly introduce
new failures.  (The good news is that this patch is already in RHEL 7.3 and
DTS 4.1, so we would not have to backport the patch there.  Unfortunately the
patch would need to be backported to DTS 4.0 however).

  Alternatively I could probably develop a much smaller patch, targeted at
just fixing this particular test case.  Less disruption, quicker to create, but
it would still leave the possibility of similar seg-faults from other sources
of corrupt debug information.

  I am not sure what the best approach is here however.  Thoughts ?

  The DWARF errors are not all the binutils fault however.  They are almost
certainly created by a combination of the compiler not separating out debug
information on a per-function basis and the linker not discarding a 
functions's debug information when it discards the function's code.  (Because
of linker garbage collection).  Anyway this is a long standing problem that
has not been properly fixed yet (anywhere, even in the mainstream development
sources), so I am not going to try to fix it for this BZ.

Cheers
  Nick
Comment 15 Nick Clifton 2016-02-26 05:48 EST
Created attachment 1130820 [details]
Prevent attempts to read an corrupt DWARF DIE value

FYI - this is the small patch that fixes just the seg-fault when reading ceph-mon.
Comment 16 Brad Hubbard 2016-02-26 07:04:47 EST
Oh wow, we have opened a can of worms with this one Nick.

With only one "reported case" of this so far I don't think it's currently worth an extensive effort but I also don't think I'm necessarily the guy to ask as I'm far from an expert in this area. There are also workarounds for this issue such as inspecting the binary in gdb, eu-addr2line. However, rhel7 is likely to be with us for quite some time.

Frank, Do you have any comments here?
Comment 17 Nick Clifton 2016-03-01 06:13:21 EST
After talking this over with Jeff Law, I think that it is safe to say that this flaw is unlikely to represent a real CVE.  Even if the DWARF debug information were to be crafted so that the read succeeds, but returns some attacker crafted value, that value cannot be used to do harm.  It is not used to allocate space, or index into an array, but rather to locate an item on an already allocated list.  And if the item is not found, an error message is produced.

So, unless someone has a strong opinion otherwise, I will check in both of the small patches attached to this BZ, once it has QA ACK.

Cheers
  Nick
Comment 18 Brad Hubbard 2016-03-01 16:46:02 EST
(In reply to Nick Clifton from comment #17)
> 
> So, unless someone has a strong opinion otherwise, I will check in both of
> the small patches attached to this BZ, once it has QA ACK.

I'm happy with whatever you decide Nick.
Comment 20 Mike McCune 2016-03-28 19:13:32 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 23 Miloš Prchlík 2016-06-24 03:46:01 EDT
Verified for build binutils-2.25.1-19.base.el7.
Comment 31 errata-xmlrpc 2016-11-03 21:55:11 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2265.html

Note You need to log in before you can comment on or make changes to this bug.