+++ This bug was initially created as a clone of Bug #1366145 +++

When dwz is run on a simple binary compiled with devtoolset-6 toolchain, it complains about section offsets not monotonically increasing:

=======
# cat test.c 
int main() { return 0; }

# rpm -qf `which gcc objdump dwz`
devtoolset-6-gcc-6.1.1-4.1.el7.x86_64
devtoolset-6-binutils-2.27-3.el7.x86_64
devtoolset-6-dwz-0.12-1.el7.x86_64
# 
# gcc test.c -g -o test
# ./test
# 
# md5sum test
dcafda5ca2f21fe103f83f50f5f49622  test
# dwz ./test
dwz: Section offsets in ./test not monotonically increasing
# md5sum test
dcafda5ca2f21fe103f83f50f5f49622  test
#
=======

When this source gets compiled with the baseos toolchain (gcc-4.8.5-9.el7, binutils-2.25.1-20.base.el7), then devtoolset-6-dwz works fine on the resulting binary.

This reminds me the prelink bug 1357016 in some way.  Applies consistently to both rhel-6 and rhel-7, all supported arches.

Although this might not be a dwz bug, I'm filing it against dwz as a start.

--- Additional comment from Jakub Jelinek on 2016-08-12 10:38:55 EDT ---

Looks like binutils bug to me.
If I link with the DTS ld, I get:
...
  [25] .bss              NOBITS          0000000000601024 001024 000004 00  WA  0   0  4
  [26] .comment          PROGBITS        0000000000000000 001024 000059 01  MS  0   0  1
  [27] .debug_aranges    PROGBITS        0000000000000000 00107d 000030 00      0   0  1
  [28] .debug_info       PROGBITS        0000000000000000 0010ad 000054 00      0   0  1
  [29] .debug_abbrev     PROGBITS        0000000000000000 001101 000037 00      0   0  1
  [30] .debug_line       PROGBITS        0000000000000000 001138 000039 00      0   0  1
  [31] .debug_str        PROGBITS        0000000000000000 001171 00005a 01  MS  0   0  1
  [32] .shstrtab         STRTAB          0000000000000000 001a28 000148 00      0   0  1
  [33] .symtab           SYMTAB          0000000000000000 0011d0 000678 18     34  53  8
  [34] .strtab           STRTAB          0000000000000000 001848 0001e0 00      0   0  1
in the test binary (note the sh_offset for .shstrtab being out of order).
Either .shstrtab should be emitted after .strtab instead of .symtab, because it is placed there, or it should use sh_offset 0x11d0 or better 0x11cb, then .symtab 0x1318 and finally .strtab 0x1990.
If I link with the system ld instead, I get sane ordering:
  [25] .bss              NOBITS          0000000000600834 000834 000004 00  WA  0   0  4
  [26] .comment          PROGBITS        0000000000000000 000834 000059 01  MS  0   0  1
  [27] .debug_aranges    PROGBITS        0000000000000000 00088d 00002f 00      0   0  1
  [28] .debug_info       PROGBITS        0000000000000000 0008bc 000042 00      0   0  1
  [29] .debug_abbrev     PROGBITS        0000000000000000 0008fe 000037 00      0   0  1
  [30] .debug_line       PROGBITS        0000000000000000 000935 000039 00      0   0  1
  [31] .debug_str        PROGBITS        0000000000000000 00096e 00005a 01  MS  0   0  1
  [32] .shstrtab         STRTAB          0000000000000000 0009c8 000148 00      0   0  1
  [33] .symtab           SYMTAB          0000000000000000 0013d0 000678 18     34  50  8
  [34] .strtab           STRTAB          0000000000000000 001a48 000234 00      0   0  1
-fuse-ld=gold with DTS binutils also works sanely:
  [26] .bss              NOBITS          0000000000402010 001010 000004 00  WA  0   0  4
  [27] .comment          PROGBITS        0000000000000000 001010 00005a 01  MS  0   0  1
  [28] .debug_info       PROGBITS        0000000000000000 00106a 000054 00      0   0  1
  [29] .debug_abbrev     PROGBITS        0000000000000000 0010be 000037 00      0   0  1
  [30] .debug_aranges    PROGBITS        0000000000000000 0010f5 000030 00      0   0  1
  [31] .debug_line       PROGBITS        0000000000000000 001125 000039 00      0   0  1
  [32] .debug_str        PROGBITS        0000000000000000 00115e 000068 01  MS  0   0  1
  [33] .note.gnu.gold-version NOTE            0000000000000000 0011c8 00001c 00      0   0  4
  [34] .symtab           SYMTAB          0000000000000000 0011e8 0003a8 18     35  22  8
  [35] .strtab           STRTAB          0000000000000000 001590 000234 00      0   0  1
  [36] .shstrtab         STRTAB          0000000000000000 0017c4 00016f 00      0   0  1
dwz isn't the only tool that expects sanely working linker.

--- Additional comment from Jakub Jelinek on 2016-08-12 12:15:07 EDT ---

I think it is 3e19fb8f990e4ce8a08f9cf2817cd9e9398648d5 https://sourceware.org/bugzilla/show_bug.cgi?id=18277 change that broke this.

--- Additional comment from Nick Clifton on 2016-08-12 12:44:09 EDT ---

Thanks for the pointer - I will look into it on Monday

--- Additional comment from Nick Clifton on 2016-08-15 12:15:11 EDT ---

Just to note that technically the linker is not doing anything wrong.  The ELF spec does not require monotonically increasing section offsets, so tools that rely upon such a feature are in fact broken.

Anyway I have tried several patches so far, but they all break something else in the section layout code.  This stuff is a mess...

--- Additional comment from Jakub Jelinek on 2016-08-15 15:21:23 EDT ---

I know it is technically valid ELF, but the sh_offset adjustments are hard enough even when they are monotonically increasing, if all the sections are laid out at random, having all tools cope with all possible section table vs. sh_offset orderings and special cases for .bss-ish and zero size sections is just too complex.  That is why dwz/prelink and other tools just give up early if they are invoked on too weird section layouts.

--- Additional comment from Jakub Jelinek on 2016-08-16 16:03 EDT ---

E.g. following patch IMHO should work (with minor needed adjustments in dozen or so tests that expect particular section order).

--- Additional comment from Nick Clifton on 2016-08-17 06:45:14 EDT ---

Hi Jakub,

> E.g. following patch IMHO should work (with minor needed adjustments in
> dozen or so tests that expect particular section order).

Doh!  That is a much better solution than the one that I was working on.

It does have one deficiency however - it does not reorder relocation sections, so these still appear out of order in the resultant binary.  Eg from the ld-x86-64/pr13947 test:

  % ../binutils/readelf -S tmpdir/dump --wide
There are 16 section headers, starting at offset 0x5b8:

Section Headers:
  [Nr] Name         Type        Address          Off    Size   ES Flg Lk Inf Al
...
  [ 6] .text        PROGBITS    0000000000000210 000210 000005 00  AX  0   0  1
  [ 7] .rela.text   RELA        0000000000000000 000530 000018 18   I 13   6  8
readelf: Warning: [ 8]: Section offsets not in a monotonically increasing order.
  [ 8] .eh_frame    PROGBITS    0000000000000218 000218 000000 00   A  0   0  8

I do not know if this will be a problem for prelink/dwz.  I am investigating to see whether I can safely extend your patch to cover these sections.

Cheers
  Nick

--- Additional comment from Jakub Jelinek on 2016-08-17 07:02:41 EDT ---

dwz/prelink only support non-ET_REL, so for those tools .rel* sections aren't a problem, and unlike the .shstrtab case, it seems as/ld has been doing that since like forever, at least looking at ET_REL files in RHEL4 confirms it.
Of course, if it isn't too hard on the linker side, adjusting even the .rel* sections wouldn't hurt, but it is much lower urgency.
Actually reading the code I wrote many years ago, it seems prelink actually has code to reorder the sections in the section table first so that it is monotonically increasing unless there are overlaps or similar bogosities, and then verifies this invariant before writing a new ELF object.
dwz doesn't have this code though, all linkers just have done a good job (until recently).  In prelink it isn't just sorting, one needs to update all the section references after the sorting.  And the comments indicate also various versions of libelf (the non-elfutils ones) have issues with non-monotonically increasing sh_offset.

--- Additional comment from Nick Clifton on 2016-08-18 03:32:46 EDT ---

Fixed in binutils-2.27-6.el7

--- Additional comment from  on 2016-10-20 09:52:25 EDT ---

Verified,
bug exists in devtoolset-6-binutils-2.27-5.el7,
bug is fixed in devtoolset-6-binutils-2.27-6.el7

Beaker log: https://url.corp.redhat.com/c1fb5a5