Bug 1311352
Summary: | objdump -S disassembly code doesn't follow /usr/lib/debug/.... conventions, so can't find sources | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Frank Ch. Eigler <fche> | ||||||
Component: | binutils | Assignee: | Nick Clifton <nickc> | ||||||
Status: | CLOSED ERRATA | QA Contact: | Miloš Prchlík <mprchlik> | ||||||
Severity: | medium | Docs Contact: | |||||||
Priority: | medium | ||||||||
Version: | 7.2 | CC: | bhubbard, fweimer, law, mcermak, mjw, mnewsome, mprchlik, ohudlick, pfrankli, vumrao | ||||||
Target Milestone: | rc | Keywords: | Patch | ||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | binutils-2.25.1-10.el7 | Doc Type: | Bug Fix | ||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | |||||||||
: | 1311494 (view as bug list) | Environment: | |||||||
Last Closed: | 2016-11-04 01:55:11 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Bug Depends On: | |||||||||
Bug Blocks: | 1311494, 1311792 | ||||||||
Attachments: |
|
Description
Frank Ch. Eigler
2016-02-24 01:16:55 UTC
(sorry, s,lib64,usr/sbin, in the 'not actually' list above; was mixing two separate experiments) (same for addr2line) The .gnu_debuglink section for this binary says: Hex dump of section '.gnu_debuglink': 0x00000000 6e736364 2e6346562 75670000 54adf413 nscd.debug..T... Hi Frank, Are the conventions for locating debug files documented somewhere ? If so, please could you point me at them ? Cheers Nick PS. Where can I find the nscd-debug rpm ? It does not appear to be part of vanilla F23... Hi Nick, The debuginfo for nscd is in the glibc-debuginfo-common package as nscd is a sub-package of glibc. This should get installed if you install the glibc-debuginfo package. If fedora is affected however, this bug should be cloned since this bug is for the rhel7 release. $ dnf provides \*/nscd.debug Repository InstallMedia has no mirror or baseurl set. Last metadata expiration check performed 0:01:10 ago on Wed Feb 24 19:56:17 2016. glibc-debuginfo-common-2.20-7.fc21.x86_64 : Debug information for package glibc Repo : @System I'm afraid I'm not aware of the documentation for debuginfo paths so I'll let Frank answer that one. I should also note that nscd is not the only package affected. This behaviour was initially noticed when working with the ceph-mon binary from Red Hat Ceph Storage. I then tried nscd and found it also was affected so I think there's a good chance many, if not all, rhel7 binaries will demonstrate this behaviour. Not to suggest this is necessarily a problem with the binaries themselves as that is yet to be determined positively. Created attachment 1130176 [details]
Add Red Hat specific search locations for debug info files
Hi Guys,
Ah - the bug does not exist in Fedora, which explains why I was confused.
Fortunately it also leads to an answer - the Fedora binutils include a
patch specifically to add extra search paths to the find_separate_debug_file
function.
The uploaded patch adds this functionality to RHEL 7.2. I cannot apply it
until I receive all the required ACKs however, but feel free to test it out
locally.
The same problem exists in RHEL 7.3. so if this BZ can be cloned for that
release then I can apply the same patch there...
Cheers
Nick
Done Nick, bz 1311494 I'll test the patch tomorrow as it's getting late for me. If it's not too much trouble could you explain how this bug came about as I'm sure this used to work on rhel6, if not on 7.1. Many thanks for your efforts. Hi Brad, > bz 1311494 Thanks. > If it's not too much trouble could you explain how this bug came about as > I'm sure this used to work on rhel6, if not on 7.1. Really ? Then I am not sure. Both 7.1 and 6.8 binutils have support for separate debuginfo files, but neither of them have the patch to search below /usr/lib/debug, so in theory they ought to be exhibiting the same problem. Maybe the installation of debuginfo files into /usr/lib/debug is a more recent change ? 64-bit RHEL6 searches below /usr/lib64/debug so maybe that is where the debuginfo files are installed ? Cheers Nick I worked out why I thought this used to work. I found the old document where I'd written about the use of "objdump -rdS" and realised that I had been running it on a binary I had compiled from source that was unstripped. Now, on to testing the patch. Hi Nick, I built a test package including your patch and it seems to work fine for the nscd binary but when I try it on ceph-mon I get the following. # objdump -rdS /usr/bin/ceph-mon &> output Segmentation fault # head output /usr/bin/ceph-mon: file format elf64-x86-64 Disassembly of section .init: 00000000005484f0 <_init>: BFD: Dwarf Error: Offset (4207821855) greater than or equal to .debug_str size (39495439). BFD: Dwarf Error: Offset (4245841173) greater than or equal to .debug_str size (39495439). BFD: Dwarf Error: Offset (3958553601) greater than or equal to .debug_str size (39495439). # tail output } bool should_gather(unsigned sub, int level) { 54afcd: 50 push %rax assert(sub < m_subsys.size()); 54afce: 48 8d 35 9b 99 3b 00 lea 0x3b999b(%rip),%rsi # 904970 <_IO_stdin_used+0x10> 54afd5: 48 8d 3d a9 99 3b 00 lea 0x3b99a9(%rip),%rdi # 904985 <_IO_stdin_used+0x25> 54afdc: ba 3e 00 00 00 mov $0x3e,%edx 54afe1: e8 5a af 26 00 callq 7b5f40 <_ZN4ceph18__ceph_assert_failEPKcS1_iS1_> BFD: Dwarf Error: Offset (1074004614) greater than or equal to .debug_str size (39495439). So it's kind of working as I can see some source output in the output but all the dwarf errors and the segfault are, of course, a show stopper. Hi Brad,
> I built a test package including your patch and it seems to work fine for
> the nscd binary but when I try it on ceph-mon I get the following.
>
> # objdump -rdS /usr/bin/ceph-mon &> output
> Segmentation fault
Darn. Can you point me at a machine where I can try this out for myself please ? (I only have 5Gb left on my root partition and that is not enough to install ceph and its debuginfo. :-(
Cheers
Nick
Hi Brad, Hmm, tricky. I can now reproduce the problem. Essentially what is happening is that corrupt DWARF information is causing the BFD library to attempt to read a LEB128 encoded value at an illegal address. The problem is, how to fix this. A while ago I spent some time hardening the BFD library so that it would catch problems like this and report the corruption, rather than performing illegal reads. The problem is that this was a very extensive patch, and backporting it in to RHEL-7.2 would take a long time, and possibly introduce new failures. (The good news is that this patch is already in RHEL 7.3 and DTS 4.1, so we would not have to backport the patch there. Unfortunately the patch would need to be backported to DTS 4.0 however). Alternatively I could probably develop a much smaller patch, targeted at just fixing this particular test case. Less disruption, quicker to create, but it would still leave the possibility of similar seg-faults from other sources of corrupt debug information. I am not sure what the best approach is here however. Thoughts ? The DWARF errors are not all the binutils fault however. They are almost certainly created by a combination of the compiler not separating out debug information on a per-function basis and the linker not discarding a functions's debug information when it discards the function's code. (Because of linker garbage collection). Anyway this is a long standing problem that has not been properly fixed yet (anywhere, even in the mainstream development sources), so I am not going to try to fix it for this BZ. Cheers Nick Created attachment 1130820 [details]
Prevent attempts to read an corrupt DWARF DIE value
FYI - this is the small patch that fixes just the seg-fault when reading ceph-mon.
Oh wow, we have opened a can of worms with this one Nick. With only one "reported case" of this so far I don't think it's currently worth an extensive effort but I also don't think I'm necessarily the guy to ask as I'm far from an expert in this area. There are also workarounds for this issue such as inspecting the binary in gdb, eu-addr2line. However, rhel7 is likely to be with us for quite some time. Frank, Do you have any comments here? After talking this over with Jeff Law, I think that it is safe to say that this flaw is unlikely to represent a real CVE. Even if the DWARF debug information were to be crafted so that the read succeeds, but returns some attacker crafted value, that value cannot be used to do harm. It is not used to allocate space, or index into an array, but rather to locate an item on an already allocated list. And if the item is not found, an error message is produced. So, unless someone has a strong opinion otherwise, I will check in both of the small patches attached to this BZ, once it has QA ACK. Cheers Nick (In reply to Nick Clifton from comment #17) > > So, unless someone has a strong opinion otherwise, I will check in both of > the small patches attached to this BZ, once it has QA ACK. I'm happy with whatever you decide Nick. This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions Verified for build binutils-2.25.1-19.base.el7. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2265.html |