Bug 1523457
| Summary: | ghc-8.2.2 build linking error on rawhide ppc64le with binutils-2.29.1 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Jens Petersen <petersen> | ||||
| Component: | binutils | Assignee: | Nick Clifton <nickc> | ||||
| Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | rawhide | CC: | aoliva, dvlasenk, haskell-devel, jakub, nickc, petersen | ||||
| Target Milestone: | --- | Keywords: | Reopened | ||||
| Target Release: | --- | ||||||
| Hardware: | ppc64le | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | binutils-2.29.1-14.fc28 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-01-28 07:05:41 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1486878, 1534526 | ||||||
| Attachments: |
|
||||||
|
Description
Jens Petersen
2017-12-08 03:03:19 UTC
I did some bissecting in mock on a ppc64le box and found that it builds okay with binutils-2.29-9.fc27 but not 2.29.1-1.fc28 or higher (at least it also fails with -5.fc28 and -6.fc28). So it seems that binutils-2.29.1 breaks ghc-8.2.2 builds on ppc64le, though ghc-8.0.2 builds itself okay. https://petersen.fedorapeople.org/copr/ghc-8.2.2-60.3.src.rpm is the srpm I am testing with. I am hoping to update F28 to ghc-8.2.2 soon if possible so this issue is blocking me from doing that. Hi Jens, Please could you capture the linker command line being generated by ghc and post that here. Even better would be if you could include a tarball of the object files being linked, so that I can attempt to reproduce the failure in a debugging environment. Cheers Nick PS. You might also find that reporting this problem on the FSF binutils bugzilla system will help as there are several PowerPC linker experts who regularly check this system. Hi Nick I ran : # "/usr/bin/ghc" -o ghc/stage1/build/tmp/ghc-stage1 -hisuf hi -osuf o -hcsuf hc -static -H32m -O -Wall -package-db libraries/bootstrapping.conf -hide-all-packages -i -ighc/. -ighc/stage1/build -Ighc/stage1/build -ighc/stage1/build/ghc/autogen -Ighc/stage1/build/ghc/autogen -optP-include -optPghc/stage1/build/ghc/autogen/cabal_macros.h -package-id base-4.9.1.0 -package-id array-0.5.1.1 -package-id bytestring-0.10.8.1 -package-id directory-1.3.0.0 -package-id process-1.4.3.0 -package-id filepath-1.4.1.1 -package-id ghc-boot-8.2.2 -package-id ghc-8.2.2 -package-id unix-2.7.2.1 -Wall -XHaskell2010 -no-hs-main -no-user-package-db -rtsopts -odir ghc/stage1/build -hidir ghc/stage1/build -stubdir ghc/stage1/build -static -H32m -O -Wall -package-db libraries/bootstrapping.conf -hide-all-packages -i -ighc/. -ighc/stage1/build -Ighc/stage1/build -ighc/stage1/build/ghc/autogen -Ighc/stage1/build/ghc/autogen -optP-include -optPghc/stage1/build/ghc/autogen/cabal_macros.h -package-id base-4.9.1.0 -package-id array-0.5.1.1 -package-id bytestring-0.10.8.1 -package-id directory-1.3.0.0 -package-id process-1.4.3.0 -package-id filepath-1.4.1.1 -package-id ghc-boot-8.2.2 -package-id ghc-8.2.2 -package-id unix-2.7.2.1 -Wall -XHaskell2010 -no-hs-main -no-user-package-db -rtsopts ghc/stage1/build/Main.o ghc/stage1/build/hschooks.o -v and output included: Created temporary directory: /tmp/ghcaf3d_0 *** C Compiler: /usr/bin/gcc -fno-stack-protector -c /tmp/ghcaf3d_0/ghc_1.c -o /tmp/ghcaf3d_0/ghc_2.o -I/usr/lib64/ghc-8.0.2/include *** C Compiler: /usr/bin/gcc -fno-stack-protector -c /tmp/ghcaf3d_0/ghc_4.s -o /tmp/ghcaf3d_0/ghc_5.o *** Linker: /usr/bin/gcc -fno-stack-protector '-Wl,--hash-size=31' -Wl,--reduce-memory-overheads -Wl,--no-as-needed -o ghc/stage1/build/tmp/ghc-stage1 -no-pie -Wl,--gc-sections ghc/stage1/build/Main.o ghc/stage1/build/hschooks.o -L/builddir/build/BUILD/ghc-8.2.2/compiler/stage1/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/terminfo/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/hoopl/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/ghci/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/transformers/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/hpc/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/template-haskell/dist-boot/build -L/usr/lib64/ghc-8.0.2/pretty-1.1.3.3 -L/builddir/build/BUILD/ghc-8.2.2/libraries/ghc-boot/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/ghc-boot-th/dist-boot/build -L/builddir/build/BUILD/ghc-8.2.2/libraries/binary/dist-boot/build -L/usr/lib64/ghc-8.0.2/containers-0.5.7.1 -L/usr/lib64/ghc-8.0.2/process-1.4.3.0 -L/usr/lib64/ghc-8.0.2/directory-1.3.0.0 -L/usr/lib64/ghc-8.0.2/unix-2.7.2.1 -L/usr/lib64/ghc-8.0.2/time-1.6.0.1 -L/usr/lib64/ghc-8.0.2/filepath-1.4.1.1 -L/usr/lib64/ghc-8.0.2/bytestring-0.10.8.1 -L/usr/lib64/ghc-8.0.2/deepseq-1.4.2.0 -L/usr/lib64/ghc-8.0.2/array-0.5.1.1 -L/usr/lib64/ghc-8.0.2/base-4.9.1.0 -L/usr/lib64/ghc-8.0.2/integer-gmp-1.0.0.1 -L/usr/lib64/ghc-8.0.2/ghc-prim-0.5.0.0 -L/usr/lib64/ghc-8.0.2/rts /tmp/ghcaf3d_0/ghc_2.o /tmp/ghcaf3d_0/ghc_5.o -Wl,-u,ghczmprim_GHCziTypes_Izh_static_info -Wl,-u,ghczmprim_GHCziTypes_Czh_static_info -Wl,-u,ghczmprim_GHCziTypes_Fzh_static_info -Wl,-u,ghczmprim_GHCziTypes_Dzh_static_info -Wl,-u,base_GHCziPtr_Ptr_static_info -Wl,-u,ghczmprim_GHCziTypes_Wzh_static_info -Wl,-u,base_GHCziInt_I8zh_static_info -Wl,-u,base_GHCziInt_I16zh_static_info -Wl,-u,base_GHCziInt_I32zh_static_info -Wl,-u,base_GHCziInt_I64zh_static_info -Wl,-u,base_GHCziWord_W8zh_static_info -Wl,-u,base_GHCziWord_W16zh_static_info -Wl,-u,base_GHCziWord_W32zh_static_info -Wl,-u,base_GHCziWord_W64zh_static_info -Wl,-u,base_GHCziStable_StablePtr_static_info -Wl,-u,ghczmprim_GHCziTypes_Izh_con_info -Wl,-u,ghczmprim_GHCziTypes_Czh_con_info -Wl,-u,ghczmprim_GHCziTypes_Fzh_con_info -Wl,-u,ghczmprim_GHCziTypes_Dzh_con_info -Wl,-u,base_GHCziPtr_Ptr_con_info -Wl,-u,base_GHCziPtr_FunPtr_con_info -Wl,-u,base_GHCziStable_StablePtr_con_info -Wl,-u,ghczmprim_GHCziTypes_False_closure -Wl,-u,ghczmprim_GHCziTypes_True_closure -Wl,-u,base_GHCziPack_unpackCString_closure -Wl,-u,base_GHCziIOziException_stackOverflow_closure -Wl,-u,base_GHCziIOziException_heapOverflow_closure -Wl,-u,base_ControlziExceptionziBase_nonTermination_closure -Wl,-u,base_GHCziIOziException_blockedIndefinitelyOnMVar_closure -Wl,-u,base_GHCziIOziException_blockedIndefinitelyOnSTM_closure -Wl,-u,base_GHCziIOziException_allocationLimitExceeded_closure -Wl,-u,base_ControlziExceptionziBase_nestedAtomically_closure -Wl,-u,base_GHCziEventziThread_blockedOnBadFD_closure -Wl,-u,base_GHCziWeak_runFinalizzerBatch_closure -Wl,-u,base_GHCziTopHandler_flushStdHandles_closure -Wl,-u,base_GHCziTopHandler_runIO_closure -Wl,-u,base_GHCziTopHandler_runNonIO_closure -Wl,-u,base_GHCziConcziIO_ensureIOManagerIsRunning_closure -Wl,-u,base_GHCziConcziIO_ioManagerCapabilitiesChanged_closure -Wl,-u,base_GHCziConcziSync_runSparks_closure -Wl,-u,base_GHCziConcziSignal_runHandlersPtr_closure -lHSghc-8.2.2 -lHSterminfo-0.4.1.0 -lHShoopl-3.10.2.2 -lHSghci-8.2.2 -lHStransformers-0.5.2.0 -lHShpc-0.6.0.3 -lHStemplate-haskell-2.12.0.0 -lHSpretty-1.1.3.3 -lHSghc-boot-8.2.2 -lHSghc-boot-th-8.2.2 -lHSbinary-0.8.5.1 -lHScontainers-0.5.7.1 -lHSprocess-1.4.3.0 -lHSdirectory-1.3.0.0 -lHSunix-2.7.2.1 -lHStime-1.6.0.1 -lHSfilepath-1.4.1.1 -lHSbytestring-0.10.8.1 -lHSdeepseq-1.4.2.0 -lHSarray-0.5.1.1 -lHSbase-4.9.1.0 -lHSinteger-gmp-1.0.0.1 -lHSghc-prim-0.5.0.0 -lHSrts -ltinfo -lrt -lutil -ldl -lpthread -lgmp -lm -lrt -ldl -lffi -lpthread It pretty easy to reproduce this in mock on a Fedora ppc64le box with: $ ssh ppc64le-test.fedorainfracloud.org # wget https://petersen.fedorapeople.org/copr/ghc-8.2.2-60.3.src.rpm # mock --old-chroot -r fedora-rawhide-ppc64le ghc-8.2.2-60.3.src.rpm (it takes about 20min to reach the linking error from a cold build, then you can go into the source tree and run make to reproduce again). Created attachment 1365908 [details]
ppc64le object files
(Even "mock --old-chroot -r fedora-rawhide-ppc64le --shell" might be enough currently.) Hi Jens, I believe that this might be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1523946 If you add "-optl -z -optl norelro" to the ghc command line then the problem goes away. So I am going to investigate fixing that BZ first, and then we can see if that also fixes your problem. Cheers Nick Thanks Hi Jens, Please could you try this binutils and let me know if it solves the problem for you: binutils-2.29.1-8.fc28 Cheers Nick Thanks a lot, Nick It looks good to me. https://koji.fedoraproject.org/koji/taskinfo?taskID=23648463 This seems to be happening again now in f28... https://koji.fedoraproject.org/koji/taskinfo?taskID=24406159 Help!! I tried to test with configuring the ghc-8.2 build with -z norelro but so far that is not helping. I tested with binutils-2.29.1-10.fc28.ppc64le (the build before relro was enabled by default for ppc64le and it built okay. https://koji.fedoraproject.org/koji/taskinfo?taskID=24412357 Hi Jens,
> I tried to test with configuring the ghc-8.2 build with -z norelro but so
> far that is not helping.
I am running some tests now. I did find that adding the "-optl -z -optl norelro"
options to the problematic ghc command worked for me. I am not sure how you enable this in the ghc.spec file however.
My biggest problem is trying to isolate a linker command line that I can use to reproduce the problem, so that I can investigate its cause...
Cheers
Nick
Hi Jens, Interesting new factoid - the problem also seems to be related to the LTO plugin. If I run the failing ghc command line with "-optl -fno-lto" added (and not the "-optl -z -optl norelo" options) then the link works. Still do not know what is actually causing the bug however... Cheers Nick Thanks, Nick, for looking into this: interesting indeed. Okay, I am not familiar with the LTO plugin. Some other datapoints: it happens when bootstrapping ghc-8.2.2 against ghc-8.0.2, but not when building ghc-8.2.2 against itself. Similarly ghc-8.0.2 building itself also seems okay: not sure why though. https://koji.fedoraproject.org/koji/taskinfo?taskID=24418947 Hi Jens, Good news: The bug is fixed in the (about to be released) 2.30 FSF binutils sources. Bad news: The sources have not been released yet, and finding the specific commit to the FSF sources that fixed the bug is going to take me *ages*. Other news: I think both LTO and relro are red herrings. The problem appears to be an instability in the code in the PowerPC linker's backend that computes the call stubs needed to access the PLT. As the linker moves sections around in memory one stub switches from containing a backwards branch to containing a forwards branch, altering its size and confusing the code which had previously allocated space for the stub. Turning off relro or LTO does not really fix the problem, it just means that different stubs to different addresses are needed, and so the bug is not triggered. Cheers Nick Hi Jens, Please try: binutils-2.29.1-14.fc28 I would love to say that I have tracked down the bug and fixed it, but it was not me. Instead I asked Alan Modra who is the guru of everything PowerPC and he proposed a small patch to fix the problem. It works in the tests I tried, so I have added it to the binutils and now it is your turn. :-) Cheers Nick PS. You may find that binutils-2.29.1-15.fc28 is available by the time you come to run your tests. This is basically the same as -14, except with binary annotations enabled. This should make no difference to you however. Thanks a lot, Nick Sure I will test it in f28-ghc as soon as newRepo runs again ;) :) Actually I can test now today in f28: https://koji.fedoraproject.org/koji/taskinfo?taskID=24486236 Okay this one might actually start building https://koji.fedoraproject.org/koji/taskinfo?taskID=24487299 But it still fails: DEBUG util.py:439: No matching package to install: 'ghc-rpm-macros-extra >= 1.8' DEBUG util.py:439: Not all dependencies satisfied DEBUG util.py:439: Error: Some packages could not be found. I hope that this is nothing to do with the binutils ? Cheers Nick Yes, doh... I have been rather busy in f28-ghc. Okay, who knows this one might get further: https://koji.fedoraproject.org/koji/taskinfo?taskID=24501387 (I think this build will not complete (due to older ghc-rpm-macros in f28 still) but it should get far enough to know that this is now hopefully fixed.) Looks good to me |