Description of problem: ghc-8.2.2 fails to build on Fedora 28 ppc64le. Version-Release number of selected component (if applicable): ghc-8.2.2 How reproducible: 100% Steps to Reproduce: 1. koji build --scratch --arch-override ppc64le f28 https://petersen.fedorapeople.org/copr/ghc-8.2.2-60.3.src.rpm Actual results: : : "/usr/bin/ar" q compiler/stage1/build/libHSghc-8.2.2.a @compiler/stage1/build/libHSghc-8.2.2.a.contents /usr/bin/ar: creating compiler/stage1/build/libHSghc-8.2.2.a "rm" -f compiler/stage1/build/libHSghc-8.2.2.a.contents "/usr/bin/ghc" -o ghc/stage1/build/tmp/ghc-stage1 -hisuf hi -osuf o -hcsuf hc -static -H32m -O -Wall -package-db libraries/bootstrapping.conf -hide-all-packages -i -ighc/. -ighc/stage1/build -Ighc/stage1/build -ighc/stage1/build/ghc/autogen -Ighc/stage1/build/ghc/autogen -optP-include -optPghc/stage1/build/ghc/autogen/cabal_macros.h -package-id base-4.9.1.0 -package-id array-0.5.1.1 -package-id bytestring-0.10.8.1 -package-id directory-1.3.0.0 -package-id process-1.4.3.0 -package-id filepath-1.4.1.1 -package-id ghc-boot-8.2.2 -package-id ghc-8.2.2 -package-id unix-2.7.2.1 -Wall -XHaskell2010 -no-hs-main -no-user-package-db -rtsopts -odir ghc/stage1/build -hidir ghc/stage1/build -stubdir ghc/stage1/build -static -H32m -O -Wall -package-db libraries/bootstrapping.conf -hide-all-packages -i -ighc/. -ighc/stage1/build -Ighc/stage1/build -ighc/stage1/build/ghc/autogen -Ighc/stage1/build/ghc/autogen -optP-include -optPghc/stage1/build/ghc/autogen/cabal_macros.h -package-id base-4.9.1.0 -package-id array-0.5.1.1 -package-id bytestring-0.10.8.1 -package-id directory-1.3.0.0 -package-id process-1.4.3.0 -package-id filepath-1.4.1.1 -package-id ghc-boot-8.2.2 -package-id ghc-8.2.2 -package-id unix-2.7.2.1 -Wall -XHaskell2010 -no-hs-main -no-user-package-db -rtsopts ghc/stage1/build/Main.o ghc/stage1/build/hschooks.o Warning: -rtsopts and -with-rtsopts have no effect with -no-hs-main. Call hs_init_ghc() from your main() function to set these options. /usr/bin/ld: stubs don't match calculated size /usr/bin/ld: can not build stubs: Invalid operation collect2: error: ld returned 1 exit status `gcc' failed in phase `Linker'. (Exit code: 1) make[1]: *** [ghc/ghc.mk:112: ghc/stage1/build/tmp/ghc-stage1] Error 1 make: *** [Makefile:125: all] Error 2 error: Bad exit status from /var/tmp/rpm-tmp.Vi44JE (%build) Expected results: No error Additional info: This does not happen on F27 ppc64le or other archs. ghc-8.0.2 builds okay on F28 ppc64le.
I did some bissecting in mock on a ppc64le box and found that it builds okay with binutils-2.29-9.fc27 but not 2.29.1-1.fc28 or higher (at least it also fails with -5.fc28 and -6.fc28). So it seems that binutils-2.29.1 breaks ghc-8.2.2 builds on ppc64le, though ghc-8.0.2 builds itself okay. https://petersen.fedorapeople.org/copr/ghc-8.2.2-60.3.src.rpm is the srpm I am testing with. I am hoping to update F28 to ghc-8.2.2 soon if possible so this issue is blocking me from doing that.
Hi Jens, Please could you capture the linker command line being generated by ghc and post that here. Even better would be if you could include a tarball of the object files being linked, so that I can attempt to reproduce the failure in a debugging environment. Cheers Nick PS. You might also find that reporting this problem on the FSF binutils bugzilla system will help as there are several PowerPC linker experts who regularly check this system.
Hi Nick I ran : # "/usr/bin/ghc" -o ghc/stage1/build/tmp/ghc-stage1 -hisuf hi -osuf o -hcsuf hc -static -H32m -O -Wall -package-db libraries/bootstrapping.conf -hide-all-packages -i -ighc/. -ighc/stage1/build -Ighc/stage1/build -ighc/stage1/build/ghc/autogen -Ighc/stage1/build/ghc/autogen -optP-include -optPghc/stage1/build/ghc/autogen/cabal_macros.h -package-id base-4.9.1.0 -package-id array-0.5.1.1 -package-id bytestring-0.10.8.1 -package-id directory-1.3.0.0 -package-id process-1.4.3.0 -package-id filepath-1.4.1.1 -package-id ghc-boot-8.2.2 -package-id ghc-8.2.2 -package-id unix-2.7.2.1 -Wall -XHaskell2010 -no-hs-main -no-user-package-db -rtsopts -odir ghc/stage1/build -hidir ghc/stage1/build -stubdir ghc/stage1/build -static -H32m -O -Wall -package-db libraries/bootstrapping.conf -hide-all-packages -i -ighc/. -ighc/stage1/build -Ighc/stage1/build -ighc/stage1/build/ghc/autogen -Ighc/stage1/build/ghc/autogen -optP-include -optPghc/stage1/build/ghc/autogen/cabal_macros.h -package-id base-4.9.1.0 -package-id array-0.5.1.1 -package-id bytestring-0.10.8.1 -package-id directory-1.3.0.0 -package-id process-1.4.3.0 -package-id filepath-1.4.1.1 -package-id ghc-boot-8.2.2 -package-id ghc-8.2.2 -package-id unix-2.7.2.1 -Wall -XHaskell2010 -no-hs-main -no-user-package-db -rtsopts ghc/stage1/build/Main.o ghc/stage1/build/hschooks.o -v and output included: Created temporary directory: /tmp/ghcaf3d_0 *** C Compiler: /usr/bin/gcc -fno-stack-protector -c /tmp/ghcaf3d_0/ghc_1.c -o /tmp/ghcaf3d_0/ghc_2.o -I/usr/lib64/ghc-8.0.2/include *** C Compiler: /usr/bin/gcc -fno-stack-protector -c /tmp/ghcaf3d_0/ghc_4.s -o /tmp/ghcaf3d_0/ghc_5.o *** Linker: /usr/bin/gcc -fno-stack-protector '-Wl,--hash-size=31' -Wl,--reduce-memory-overheads -Wl,--no-as-needed -o ghc/stage1/build/tmp/ghc-stage1 -no-pie -Wl,--gc-sections ghc/stage1/build/Main.o ghc/stage1/build/hschooks.o -L/builddir/build/BUILD/ghc-8.2.2/compiler/stage1/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/terminfo/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/hoopl/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/ghci/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/transformers/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/hpc/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/template-haskell/dist-boot/build -L/usr/lib64/ghc-8.0.2/pretty-1.1.3.3 -L/builddir/build/BUILD/ghc-8.2.2/libraries/ghc-boot/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/ghc-boot-th/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/binary/dist-boot/build -L/usr/lib64/ghc-8.0.2/containers-0.5.7.1 -L/usr/lib64/ghc-8.0.2/process-1.4.3.0 -L/usr/lib64/ghc-8.0.2/directory-1.3.0.0 -L/usr/lib64/ghc-8.0.2/unix-2.7.2.1 -L/usr/lib64/ghc-8.0.2/time-1.6.0.1 -L/usr/lib64/ghc-8.0.2/filepath-1.4.1.1 -L/usr/lib64/ghc-8.0.2/bytestring-0.10.8.1 -L/usr/lib64/ghc-8.0.2/deepseq-1.4.2.0 -L/usr/lib64/ghc-8.0.2/array-0.5.1.1 -L/usr/lib64/ghc-8.0.2/base-4.9.1.0 -L/usr/lib64/ghc-8.0.2/integer-gmp-1.0.0.1 -L/usr/lib64/ghc-8.0.2/ghc-prim-0.5.0.0 -L/usr/lib64/ghc-8.0.2/rts /tmp/ghcaf3d_0/ghc_2.o /tmp/ghcaf3d_0/ghc_5.o -Wl,-u,ghczmprim_GHCziTypes_Izh_static_info -Wl,-u,ghczmprim_GHCziTypes_Czh_static_info -Wl,-u,ghczmprim_GHCziTypes_Fzh_static_info -Wl,-u,ghczmprim_GHCziTypes_Dzh_static_info -Wl,-u,base_GHCziPtr_Ptr_static_info -Wl,-u,ghczmprim_GHCziTypes_Wzh_static_info -Wl,-u,base_GHCziInt_I8zh_static_info -Wl,-u,base_GHCziInt_I16zh_static_info -Wl,-u,base_GHCziInt_I32zh_static_info -Wl,-u,base_GHCziInt_I64zh_static_info -Wl,-u,base_GHCziWord_W8zh_static_info -Wl,-u,base_GHCziWord_W16zh_static_info -Wl,-u,base_GHCziWord_W32zh_static_info -Wl,-u,base_GHCziWord_W64zh_static_info -Wl,-u,base_GHCziStable_StablePtr_static_info -Wl,-u,ghczmprim_GHCziTypes_Izh_con_info -Wl,-u,ghczmprim_GHCziTypes_Czh_con_info -Wl,-u,ghczmprim_GHCziTypes_Fzh_con_info -Wl,-u,ghczmprim_GHCziTypes_Dzh_con_info -Wl,-u,base_GHCziPtr_Ptr_con_info -Wl,-u,base_GHCziPtr_FunPtr_con_info -Wl,-u,base_GHCziStable_StablePtr_con_info -Wl,-u,ghczmprim_GHCziTypes_False_closure -Wl,-u,ghczmprim_GHCziTypes_True_closure -Wl,-u,base_GHCziPack_unpackCString_closure -Wl,-u,base_GHCziIOziException_stackOverflow_closure -Wl,-u,base_GHCziIOziException_heapOverflow_closure -Wl,-u,base_ControlziExceptionziBase_nonTermination_closure -Wl,-u,base_GHCziIOziException_blockedIndefinitelyOnMVar_closure -Wl,-u,base_GHCziIOziException_blockedIndefinitelyOnSTM_closure -Wl,-u,base_GHCziIOziException_allocationLimitExceeded_closure -Wl,-u,base_ControlziExceptionziBase_nestedAtomically_closure -Wl,-u,base_GHCziEventziThread_blockedOnBadFD_closure -Wl,-u,base_GHCziWeak_runFinalizzerBatch_closure -Wl,-u,base_GHCziTopHandler_flushStdHandles_closure -Wl,-u,base_GHCziTopHandler_runIO_closure -Wl,-u,base_GHCziTopHandler_runNonIO_closure -Wl,-u,base_GHCziConcziIO_ensureIOManagerIsRunning_closure -Wl,-u,base_GHCziConcziIO_ioManagerCapabilitiesChanged_closure -Wl,-u,base_GHCziConcziSync_runSparks_closure -Wl,-u,base_GHCziConcziSignal_runHandlersPtr_closure -lHSghc-8.2.2 -lHSterminfo-0.4.1.0 -lHShoopl-3.10.2.2 -lHSghci-8.2.2 -lHStransformers-0.5.2.0 -lHShpc-0.6.0.3 -lHStemplate-haskell-2.12.0.0 -lHSpretty-1.1.3.3 -lHSghc-boot-8.2.2 -lHSghc-boot-th-8.2.2 -lHSbinary-0.8.5.1 -lHScontainers-0.5.7.1 -lHSprocess-1.4.3.0 -lHSdirectory-1.3.0.0 -lHSunix-2.7.2.1 -lHStime-1.6.0.1 -lHSfilepath-1.4.1.1 -lHSbytestring-0.10.8.1 -lHSdeepseq-1.4.2.0 -lHSarray-0.5.1.1 -lHSbase-4.9.1.0 -lHSinteger-gmp-1.0.0.1 -lHSghc-prim-0.5.0.0 -lHSrts -ltinfo -lrt -lutil -ldl -lpthread -lgmp -lm -lrt -ldl -lffi -lpthread
It pretty easy to reproduce this in mock on a Fedora ppc64le box with: $ ssh ppc64le-test.fedorainfracloud.org # wget https://petersen.fedorapeople.org/copr/ghc-8.2.2-60.3.src.rpm # mock --old-chroot -r fedora-rawhide-ppc64le ghc-8.2.2-60.3.src.rpm (it takes about 20min to reach the linking error from a cold build, then you can go into the source tree and run make to reproduce again).
Created attachment 1365908 [details] ppc64le object files
(Even "mock --old-chroot -r fedora-rawhide-ppc64le --shell" might be enough currently.)
Hi Jens, I believe that this might be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1523946 If you add "-optl -z -optl norelro" to the ghc command line then the problem goes away. So I am going to investigate fixing that BZ first, and then we can see if that also fixes your problem. Cheers Nick
Thanks
Hi Jens, Please could you try this binutils and let me know if it solves the problem for you: binutils-2.29.1-8.fc28 Cheers Nick
Thanks a lot, Nick It looks good to me. https://koji.fedoraproject.org/koji/taskinfo?taskID=23648463
This seems to be happening again now in f28... https://koji.fedoraproject.org/koji/taskinfo?taskID=24406159 Help!!
I tried to test with configuring the ghc-8.2 build with -z norelro but so far that is not helping.
I tested with binutils-2.29.1-10.fc28.ppc64le (the build before relro was enabled by default for ppc64le and it built okay. https://koji.fedoraproject.org/koji/taskinfo?taskID=24412357
Hi Jens, > I tried to test with configuring the ghc-8.2 build with -z norelro but so > far that is not helping. I am running some tests now. I did find that adding the "-optl -z -optl norelro" options to the problematic ghc command worked for me. I am not sure how you enable this in the ghc.spec file however. My biggest problem is trying to isolate a linker command line that I can use to reproduce the problem, so that I can investigate its cause... Cheers Nick
Hi Jens, Interesting new factoid - the problem also seems to be related to the LTO plugin. If I run the failing ghc command line with "-optl -fno-lto" added (and not the "-optl -z -optl norelo" options) then the link works. Still do not know what is actually causing the bug however... Cheers Nick
Thanks, Nick, for looking into this: interesting indeed. Okay, I am not familiar with the LTO plugin. Some other datapoints: it happens when bootstrapping ghc-8.2.2 against ghc-8.0.2, but not when building ghc-8.2.2 against itself. Similarly ghc-8.0.2 building itself also seems okay: not sure why though. https://koji.fedoraproject.org/koji/taskinfo?taskID=24418947
Hi Jens, Good news: The bug is fixed in the (about to be released) 2.30 FSF binutils sources. Bad news: The sources have not been released yet, and finding the specific commit to the FSF sources that fixed the bug is going to take me *ages*. Other news: I think both LTO and relro are red herrings. The problem appears to be an instability in the code in the PowerPC linker's backend that computes the call stubs needed to access the PLT. As the linker moves sections around in memory one stub switches from containing a backwards branch to containing a forwards branch, altering its size and confusing the code which had previously allocated space for the stub. Turning off relro or LTO does not really fix the problem, it just means that different stubs to different addresses are needed, and so the bug is not triggered. Cheers Nick
Hi Jens, Please try: binutils-2.29.1-14.fc28 I would love to say that I have tracked down the bug and fixed it, but it was not me. Instead I asked Alan Modra who is the guru of everything PowerPC and he proposed a small patch to fix the problem. It works in the tests I tried, so I have added it to the binutils and now it is your turn. :-) Cheers Nick PS. You may find that binutils-2.29.1-15.fc28 is available by the time you come to run your tests. This is basically the same as -14, except with binary annotations enabled. This should make no difference to you however.
Thanks a lot, Nick Sure I will test it in f28-ghc as soon as newRepo runs again ;) :)
Actually I can test now today in f28: https://koji.fedoraproject.org/koji/taskinfo?taskID=24486236
https://koji.fedoraproject.org/koji/taskinfo?taskID=24486824
Okay this one might actually start building https://koji.fedoraproject.org/koji/taskinfo?taskID=24487299
But it still fails: DEBUG util.py:439: No matching package to install: 'ghc-rpm-macros-extra >= 1.8' DEBUG util.py:439: Not all dependencies satisfied DEBUG util.py:439: Error: Some packages could not be found. I hope that this is nothing to do with the binutils ? Cheers Nick
Yes, doh... I have been rather busy in f28-ghc. Okay, who knows this one might get further: https://koji.fedoraproject.org/koji/taskinfo?taskID=24501387 (I think this build will not complete (due to older ghc-rpm-macros in f28 still) but it should get far enough to know that this is now hopefully fixed.)
Looks good to me