Bug 1215546

I am not sure how to capture the command line used to invoke gold.
"gcc -v -o conftest -fuse-ld=gold conftest.c" does not seem to output
an explicit ld.gold invocation.

Created attachment 1019235 [details]
aarch64 object file

conftest.o captured with "gcc -save-temps -o conftest -fuse-ld=gold conftest.c"

Here is the gcc -v output anyway for good measure:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-redhat-linux/4.9.2/lto-wrapper
Target: aarch64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.9.2-20150212/obj-aarch64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.9.2-20150212/obj-aarch64-redhat-linux/cloog-install --enable-gnu-indirect-function --build=aarch64-redhat-linux
Thread model: posix
gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC) 
COLLECT_GCC_OPTIONS='-v' '-o' 'conftest' '-fuse-ld=gold' '-mlittle-endian' '-mabi=lp64'
 /usr/libexec/gcc/aarch64-redhat-linux/4.9.2/cc1 -quiet -v conftest.c -quiet -dumpbase conftest.c -mlittle-endian -mabi=lp64 -auxbase conftest -version -fuse-ld=gold -o /tmp/ccMUpatn.s
GNU C (GCC) version 4.9.2 20150212 (Red Hat 4.9.2-6) (aarch64-redhat-linux)
	compiled by GNU C version 4.9.2 20150212 (Red Hat 4.9.2-6), GMP version 6.0.0, MPFR version 3.1.2, MPC version 1.0.2
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/usr/lib/gcc/aarch64-redhat-linux/4.9.2/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../../aarch64-redhat-linux/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/aarch64-redhat-linux/4.9.2/include
 /usr/local/include
 /usr/include
End of search list.
GNU C (GCC) version 4.9.2 20150212 (Red Hat 4.9.2-6) (aarch64-redhat-linux)
	compiled by GNU C version 4.9.2 20150212 (Red Hat 4.9.2-6), GMP version 6.0.0, MPFR version 3.1.2, MPC version 1.0.2
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 9d00026f4ad4c5ca24956658dd9d77f0
COLLECT_GCC_OPTIONS='-v' '-o' 'conftest' '-fuse-ld=gold' '-mlittle-endian' '-mabi=lp64'
 as -v -EL -mabi=lp64 -o /tmp/ccQTdQSB.o /tmp/ccMUpatn.s
GNU assembler version 2.25 (aarch64-redhat-linux) using BFD version version 2.25-7.1.fc23
COMPILER_PATH=/usr/libexec/gcc/aarch64-redhat-linux/4.9.2/:/usr/libexec/gcc/aarch64-redhat-linux/4.9.2/:/usr/libexec/gcc/aarch64-redhat-linux/:/usr/lib/gcc/aarch64-redhat-linux/4.9.2/:/usr/lib/gcc/aarch64-redhat-linux/
LIBRARY_PATH=/usr/lib/gcc/aarch64-redhat-linux/4.9.2/:/usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-o' 'conftest' '-fuse-ld=gold' '-mlittle-endian' '-mabi=lp64'
 /usr/libexec/gcc/aarch64-redhat-linux/4.9.2/collect2 -plugin /usr/libexec/gcc/aarch64-redhat-linux/4.9.2/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/aarch64-redhat-linux/4.9.2/lto-wrapper -plugin-opt=-fresolution=/tmp/ccyt2r4Q.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -dynamic-linker /lib/ld-linux-aarch64.so.1 -X -EL -maarch64linux -fuse-ld=gold -o conftest /usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../../lib64/crt1.o /usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../../lib64/crti.o /usr/lib/gcc/aarch64-redhat-linux/4.9.2/crtbegin.o -L/usr/lib/gcc/aarch64-redhat-linux/4.9.2 -L/usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../.. /tmp/ccQTdQSB.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/aarch64-redhat-linux/4.9.2/crtend.o /usr/lib/gcc/aarch64-redhat-linux/4.9.2/../../../../lib64/crtn.o

Hi Jens,

  Is there an aarch64 machine I can log in to ?  I have been trying to reproduce the problem using a cross built toolchain, but I have not managed to make gold crash. :-(

Cheers
  Nick

Nick, let me send you an email.

Summary of discoveries so far:

  * Cannot reproduce the problem using a cross-built toolchain.
  * Can reproduce using a native toolchain on a Fedora 21 machine.
  * Problem exists both with rawhide binutils (based on FSF binutils 2.25) and FSF mainline development sources.
  * Problem exists only when linking with gold, not with ld.
  * A binary linked using gold running on an x86_64 host running Fedora 21 will run.

There appear to be a couple of possibilities for the cause of the problem:

  * A bug in Fedora 21 gcc means that compiling gold natively on aarch64 results in a subtly broken gold executable.

  * A bug in the gold sources is only being triggered when linking natively.

I am still investigating, but it is going to take some time.

Cheers
  Nick

Thank you for looking into this.

(In reply to Nick Clifton from comment #6)
>   * Can reproduce using a native toolchain on a Fedora 21 machine.

Note that it also happens in Rawhide, which has gcc-5.0.1-0.1.fc23.aarch64.

eg http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=2972494

binutils-2.25-8.fc23 was also built with gcc-5.0.1-0.1.fc23.aarch64.

Hi Guys,

> Summary of discoveries so far:
> 
>   * A binary linked using gold running on an x86_64 host running Fedora 21
> will run.

Scratch that.  The problem still exists with cross built executables.  The problem goes away however if you build *static* executables, either natively or with a cross-toolchain.  (I was mistakenly building a static version of conftest with my cross tools, which is why I thought cross-building worked).

So that at least suggests a temporary workaround - building a static version of ghc.

Meanwhile, now that I can compile test binaries locally I might be able to get a bit further investigating the real problem.

Does anybody know how to persuade ld-linux-aarch64.so.1 to tell me why it refuses to load a binary ?  All I get is a non-zero return value, but no error message of any kind.

Cheers
  Nick

FYI, I have reported this bug upstream:

https://sourceware.org/bugzilla/show_bug.cgi?id=18365

Created attachment 1022542 [details]
Working dynamic conftest executable

Hi Guys,

  Right - I am officially out of my depth here.  Gold *is* creating working dynamic executables.  Just not ones that will work on mustang-02.  If you read the FSF bugzilla PR you will see that Han Shenan was able to run the conftest executable created by gold on mustang-02, and he was able to link and run his own executable on his machine.  But when I try to run that same executable on mustang-02 it fails.  So there must be something about our machine that is different from his, something that prevents dynamic executables from working.

  I do not know enough about loaders to find out what is going wrong.  We need a glibc expert to tell us why ld.so does not like the conftest executables, and then maybe we can find a solution.

Cheers
  Nick

Just a wild guess, the pagesize might be different (or size of virtual address space, but the latter hopefully shouldn't matter).

Hi Jakub,

> Just a wild guess, the pagesize might be different

Is there some way to test this ?  

The working ld-produced dynamic binary and the non-working gold-produced dynamic binary both have an alignment of 0x1000 for the LOAD segments.

What do we know about the pagesize requirement of the mustang boards ?

Cheers
  Nick

PS.  I thought that it might be an executable stack problem, since Han's conftest binary has a GNU_STACK segment that has RWE flags.  But the non-working version produced locally on mustang-02 only has RW flags, so that is not it. :-(

0x1000 PT_LOAD alignment on an architecture that allows 4KB and 64KB page sizes sounds just wrong.
See https://www.kernel.org/doc/Documentation/arm64/memory.txt
Easy way to test would be
getconf PAGESIZE
I'd say.

elfnn-aarch64.c:#define ELF_MAXPAGESIZE			0x10000
elfnn-aarch64.c:#define ELF_MINPAGESIZE			0x1000
elfnn-aarch64.c:#define ELF_COMMONPAGESIZE		0x1000

looks good to me (though I hope that for Fedora/RHEL where 64KB page size is used we patch ELF_COMMONPAGESIZE to 64KB).

(In reply to Nick Clifton from comment #8)
> The problem still exists with cross built executables.  The
> problem goes away however if you build *static* executables, either natively
> or with a cross-toolchain.  (I was mistakenly building a static version of
> conftest with my cross tools, which is why I thought cross-building worked).

Ah, I forgot to comment about this:

> So that at least suggests a temporary workaround - building a static version
> of ghc.

That's right - I am actually building ghc on aarch64 with Haskell static linking to workaround a problem with stdout getting lost from
dynamically linked executables.

(In reply to Nick Clifton from comment #10)
>   Right - I am officially out of my depth here.  Gold *is* creating working
> dynamic executables.  Just not ones that will work on mustang-02.  If you
> read the FSF bugzilla PR you will see that Han Shenan was able to run the
> conftest executable created by gold on mustang-02, and he was able to link
> and run his own executable on his machine.  But when I try to run that same
> executable on mustang-02 it fails.  So there must be something about our
> machine that is different from his, something that prevents dynamic
> executables from working.

Okay this also agrees with my ghc friend who is running Debian on aarch64
and hasn't seen this problem either: so I agree that this seems to be
Fedora specific.

Any more headway with this?

Hi Jens,

  This problem *might* be fixed by binutils-2.25-10.fc23 and/or binutils-2.25-8.fc22.  Please give one of them a try and let me know if gold really is working now.

Cheers
  Nick

Yes, I tested binutils-2.25-10.fc23 with my ghc-7.10.1.20150511-1.fc21.src.rpm
and it went through configure file.  So looks good to me.

For the record the scratch build:

http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=3032351

(ignore the final filelist errors that is due to ghc-rpm-macros
needing tweaks for ghc-7.10.)

It looks good to me still: also fixes bug 1195231.

Can binutils-2.25-8.fc22 be pushed to updates-testing?

binutils-2.25-8.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/binutils-2.25-8.fc22

Package binutils-2.25-8.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing binutils-2.25-8.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-10051/binutils-2.25-8.fc22
then log in and leave karma (feedback).

binutils-2.25-8.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.