(sorry, s,lib64,usr/sbin, in the 'not actually' list above; was mixing two separate experiments)

(same for addr2line)

The .gnu_debuglink section for this binary says:

Hex dump of section '.gnu_debuglink':
  0x00000000 6e736364 2e6346562 75670000 54adf413   nscd.debug..T...

Hi Frank,

  Are the conventions for locating debug files documented somewhere ?  If so, please could you point me at them ?

Cheers
  Nick

PS.  Where can I find the nscd-debug rpm ?  It does not appear to be part of vanilla F23...

Hi Nick, The debuginfo for nscd is in the glibc-debuginfo-common package as nscd is a sub-package of glibc. This should get installed if you install the glibc-debuginfo package. If fedora is affected however, this bug should be cloned since this bug is for the rhel7 release.

$ dnf provides \*/nscd.debug
Repository InstallMedia has no mirror or baseurl set.                                 
Last metadata expiration check performed 0:01:10 ago on Wed Feb 24 19:56:17 2016.
glibc-debuginfo-common-2.20-7.fc21.x86_64 : Debug information for package glibc
Repo        : @System

I'm afraid I'm not aware of the documentation for debuginfo paths so I'll let Frank answer that one.

I should also note that nscd is not the only package affected. This behaviour was initially noticed when working with the ceph-mon binary from Red Hat Ceph Storage. I then tried nscd and found it also was affected so I think there's a good chance many, if not all, rhel7 binaries will demonstrate this behaviour. Not to suggest this is necessarily a problem with the binaries themselves as that is yet to be determined positively.

Created attachment 1130176 [details]
Add Red Hat specific search locations for debug info files

Hi Guys,

  Ah - the bug does not exist in Fedora, which explains why I was confused.
  Fortunately it also leads to an answer - the Fedora binutils include a 
  patch specifically to add extra search paths to the find_separate_debug_file
  function.

  The uploaded patch adds this functionality to RHEL 7.2.  I cannot apply it
  until I receive all the required ACKs however, but feel free to test it out
  locally.

  The same problem exists in RHEL 7.3. so if this BZ can be cloned for that
  release then I can apply the same patch there...

Cheers
  Nick

Done Nick,

bz 1311494

I'll test the patch tomorrow as it's getting late for me. 

If it's not too much trouble could you explain how this bug came about as I'm sure this used to work on rhel6, if not on 7.1.

Many thanks for your efforts.

Hi Brad,

> bz 1311494

Thanks.

> If it's not too much trouble could you explain how this bug came about as
> I'm sure this used to work on rhel6, if not on 7.1.

Really ?  Then I am not sure.  Both 7.1 and 6.8 binutils have support for 
separate debuginfo files, but neither of them have the patch to search 
below /usr/lib/debug, so in theory they ought to be exhibiting the same 
problem.

Maybe the installation of debuginfo files into /usr/lib/debug is a more recent
change ?  64-bit RHEL6 searches below /usr/lib64/debug so maybe that is where
the debuginfo files are installed ?

Cheers
  Nick

I worked out why I thought this used to work.

I found the old document where I'd written about the use of "objdump -rdS" and realised that I had been running it on a binary I had compiled from source that was unstripped.

Now, on to testing the patch.

Hi Nick,

I built a test package including your patch and it seems to work fine for the nscd binary but when I try it on ceph-mon I get the following.

# objdump -rdS /usr/bin/ceph-mon &> output
Segmentation fault

# head output

/usr/bin/ceph-mon:     file format elf64-x86-64


Disassembly of section .init:

00000000005484f0 <_init>:
BFD: Dwarf Error: Offset (4207821855) greater than or equal to .debug_str size (39495439).
BFD: Dwarf Error: Offset (4245841173) greater than or equal to .debug_str size (39495439).
BFD: Dwarf Error: Offset (3958553601) greater than or equal to .debug_str size (39495439).

# tail output
  }

  bool should_gather(unsigned sub, int level) {
  54afcd:       50                      push   %rax
    assert(sub < m_subsys.size());
  54afce:       48 8d 35 9b 99 3b 00    lea    0x3b999b(%rip),%rsi        # 904970 <_IO_stdin_used+0x10>
  54afd5:       48 8d 3d a9 99 3b 00    lea    0x3b99a9(%rip),%rdi        # 904985 <_IO_stdin_used+0x25>
  54afdc:       ba 3e 00 00 00          mov    $0x3e,%edx
  54afe1:       e8 5a af 26 00          callq  7b5f40 <_ZN4ceph18__ceph_assert_failEPKcS1_iS1_>
BFD: Dwarf Error: Offset (1074004614) greater than or equal to .debug_str size (39495439).

So it's kind of working as I can see some source output in the output but all the dwarf errors and the segfault are, of course, a show stopper.

Hi Brad,

> I built a test package including your patch and it seems to work fine for
> the nscd binary but when I try it on ceph-mon I get the following.
> 
> # objdump -rdS /usr/bin/ceph-mon &> output
> Segmentation fault

Darn.  Can you point me at a machine where I can try this out for myself please ?  (I only have 5Gb left on my root partition and that is not enough to install ceph and its debuginfo.  :-(

Cheers
  Nick

Hi Brad,

  Hmm, tricky.  I can now reproduce the problem.  Essentially what is happening
is that corrupt DWARF information is causing the BFD library to attempt to read
a LEB128 encoded value at an illegal address.  The problem is, how to fix this.

  A while ago I spent some time hardening the BFD library so that it would
catch problems like this and report the corruption, rather than performing
illegal reads.  The problem is that this was a very extensive patch, and
backporting it in to RHEL-7.2 would take a long time, and possibly introduce
new failures.  (The good news is that this patch is already in RHEL 7.3 and
DTS 4.1, so we would not have to backport the patch there.  Unfortunately the
patch would need to be backported to DTS 4.0 however).

  Alternatively I could probably develop a much smaller patch, targeted at
just fixing this particular test case.  Less disruption, quicker to create, but
it would still leave the possibility of similar seg-faults from other sources
of corrupt debug information.

  I am not sure what the best approach is here however.  Thoughts ?

  The DWARF errors are not all the binutils fault however.  They are almost
certainly created by a combination of the compiler not separating out debug
information on a per-function basis and the linker not discarding a 
functions's debug information when it discards the function's code.  (Because
of linker garbage collection).  Anyway this is a long standing problem that
has not been properly fixed yet (anywhere, even in the mainstream development
sources), so I am not going to try to fix it for this BZ.

Cheers
  Nick

Created attachment 1130820 [details]
Prevent attempts to read an corrupt DWARF DIE value

FYI - this is the small patch that fixes just the seg-fault when reading ceph-mon.

Oh wow, we have opened a can of worms with this one Nick.

With only one "reported case" of this so far I don't think it's currently worth an extensive effort but I also don't think I'm necessarily the guy to ask as I'm far from an expert in this area. There are also workarounds for this issue such as inspecting the binary in gdb, eu-addr2line. However, rhel7 is likely to be with us for quite some time.

Frank, Do you have any comments here?

After talking this over with Jeff Law, I think that it is safe to say that this flaw is unlikely to represent a real CVE.  Even if the DWARF debug information were to be crafted so that the read succeeds, but returns some attacker crafted value, that value cannot be used to do harm.  It is not used to allocate space, or index into an array, but rather to locate an item on an already allocated list.  And if the item is not found, an error message is produced.

So, unless someone has a strong opinion otherwise, I will check in both of the small patches attached to this BZ, once it has QA ACK.

Cheers
  Nick

(In reply to Nick Clifton from comment #17)
> 
> So, unless someone has a strong opinion otherwise, I will check in both of
> the small patches attached to this BZ, once it has QA ACK.

I'm happy with whatever you decide Nick.

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Verified for build binutils-2.25.1-19.base.el7.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2265.html