Description of problem: Clang is broken on armv7hl architecture. Version-Release number of selected component (if applicable): clang 11.0.1-4.fc34 gcc 11.0.0-0.16.fc34 How reproducible: Always. Steps to Reproduce: 1. Try to build nheko package in Rawhide for armv7hl. 2. 3. Actual results: CMake Warning (dev) at /usr/share/cmake/Modules/GNUInstallDirs.cmake:223 (message): Unable to determine default CMAKE_INSTALL_LIBDIR directory because no target architecture is known. Please enable at least one language before including GNUInstallDirs. Call Stack (most recent call first): CMakeLists.txt:75 (include) This warning is for project developers. Use -Wno-dev to suppress it. -- The CXX compiler identification is Clang 11.0.1 -- The C compiler identification is Clang 11.0.1 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - failed -- Check for working CXX compiler: /usr/bin/clang++ -- Check for working CXX compiler: /usr/bin/clang++ - broken CMake Error at /usr/share/cmake/Modules/CMakeTestCXXCompiler.cmake:59 (message): The C++ compiler "/usr/bin/clang++" is not able to compile a simple test program. It fails with the following output: Change Dir: /builddir/build/BUILD/nheko-0.8.0/armv7hl-redhat-linux-gnueabi/CMakeFiles/CMakeTmp Run Build Command(s):/usr/bin/ninja-build cmTC_87320 && [1/2] Building CXX object CMakeFiles/cmTC_87320.dir/testCXXCompiler.cxx.o [2/2] Linking CXX executable cmTC_87320 FAILED: cmTC_87320 : && /usr/bin/clang++ -O2 -flto -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS --config /usr/lib/rpm/redhat/redhat-hardened-clang.cfg -fstack-protector-strong -march=armv7-a -mfpu=vfpv3-d16 -mtune=generic-armv7-a -mabi=aapcs-linux -mfloat-abi=hard -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -flto CMakeFiles/cmTC_87320.dir/testCXXCompiler.cxx.o -o cmTC_87320 && : clang-11: error: unable to execute command: Segmentation fault (core dumped) clang-11: error: linker command failed due to signal (use -v to see invocation) ninja: build stopped: subcommand failed. CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:80 (project) -- Configuring incomplete, errors occurred! See also "/builddir/build/BUILD/nheko-0.8.0/armv7hl-redhat-linux-gnueabi/CMakeFiles/CMakeOutput.log". See also "/builddir/build/BUILD/nheko-0.8.0/armv7hl-redhat-linux-gnueabi/CMakeFiles/CMakeError.log". RPM build errors: error: Bad exit status from /var/tmp/rpm-tmp.zRVNb8 (%build) Bad exit status from /var/tmp/rpm-tmp.zRVNb8 (%build) Child return code was: 1 Expected results: Successful build. Additional info: Koji task: https://koji.fedoraproject.org/koji/taskinfo?taskID=60156825
I can reproduce this failure locally using mock's forcearch feature: fedpkg clone nheko cd nheko fedpkg mockbuild --root fedora-rawhide-armhfp
The segmentation fault is actually coming from the linker (ld.bfd) and not clang. I haven't been able to figure out how to get a stacktrace inside QEMU, so I can't tell if it's a bug in ld.bfd or a bug in the LLVM gold plugin. There are 2 ways you can work around this issue: 1. If you want to continue using LTO, you can use lld as the linker. This means adding BuildRequires: lld and updating the linker flags, like this: %global build_ldflags %(echo %{build_ldflags} -fuse-ld=lld) 2. If you want to keep using ld.bfd, you can just disable LTO: %global _lto_cflags %{nil}"
Added a temporary workaround to the nheko package: %ifarch %{arm} %global _lto_cflags %{nil} %endif
(In reply to Tom Stellard from comment #2) > The segmentation fault is actually coming from the linker (ld.bfd) and not > clang. I haven't been able to figure out how to get a stacktrace inside > QEMU, so I can't tell if it's a bug in ld.bfd or a bug in the LLVM gold > plugin. Given that compiling without LTO enabled makes the bug go away, I would guess that the problem is the gold plugin. This is actually part of gcc, rather than the binutils, so maybe the recent update of gcc to 11.0.0-0.6 might be the cause. But it could also be the gold linker's use of the plugin that is the cause. There is a scratch build available of a new binutils built from the forthcoming 2.36 release which you might like to try. (Note - this scratch build is not going to be turned into an official update of the rawhide binutils until after the F34 branch is created). https://koji.fedoraproject.org/koji/taskinfo?taskID=60208987
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle. Changing version to 34.
> The segmentation fault is actually coming from the linker (ld.bfd) and not clang. I haven't been able to figure out how to get a stacktrace inside QEMU, so I can't tell if it's a bug in ld.bfd or a bug in the LLVM gold plugin. There are 2 ways you can work around this issue: @Tom Stellard Yon might be able to ask someone using a native armv7 (ARM 32-bit) machine in Fedora. See https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers
*** Bug 1943738 has been marked as a duplicate of this bug. ***
I was able to get a backtrace for the crash: #0 elf32_arm_output_arch_local_syms (output_bfd=output_bfd@entry=0xff2a7188, info=info@entry=0xff2a7188, flaginfo=flaginfo@entry=0xffee6014, func=<optimized out>) at elf32-arm.c:18291 18291 if (local_iplt[i] != NULL (gdb) bt #0 elf32_arm_output_arch_local_syms (output_bfd=output_bfd@entry=0xff2a7188, info=info@entry=0xff2a7188, flaginfo=flaginfo@entry=0xffee6014, func=<optimized out>) at elf32-arm.c:18291 #1 0xff5ea8b8 in bfd_elf_final_link (abfd=<optimized out>, info=<optimized out>) at elflink.c:12828 #2 0xff5b8878 in elf32_arm_final_link (abfd=0xff2a7188, info=0xfffeebb8 <link_info>) at elf32-arm.c:13745 #3 0xffeef090 in ldwrite () at ldwrite.c:545 #4 main (argc=<optimized out>, argv=<optimized out>) at ./ldmain.c:512
(In reply to Tom Stellard from comment #8) Hi Tom, Which version of the binutils was installed ? > #0 elf32_arm_output_arch_local_syms > (output_bfd=output_bfd@entry=0xff2a7188, info=info@entry=0xff2a7188, > flaginfo=flaginfo@entry=0xffee6014, func=<optimized out>) > at elf32-arm.c:18291 > 18291 if (local_iplt[i] != NULL Are you able to collect the object files and libraries involved in this link ? And the linker command line too, if possible. Then I can attempt to reproduce the failure. Cheers Nick
Created attachment 1775725 [details] Failing object file The object file is actually LLVM bitcode (used for LTO), so I don't know if this helpful, but you can reproduce with this command: /usr/bin/ld -o main /usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/../../../../lib/crt1.o /usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/../../../../lib/crti.o /usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/crtbegin.o -L/usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11 -L/usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/../../../../lib -L/usr/bin/../lib -L/lib/../lib -L/usr/lib/../lib -L/usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib -plugin /usr/bin/../lib/LLVMgold.so clang-lto-bfd-arm-crash.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/crtend.o /usr/lib/gcc/armv7hl-redhat-linux-gnueabi/11/../../../../lib/crtn.o
By using slightly different input files, I was able to get bfd to abort(), maybe this is more helpful: <mock-chroot> sh-5.1# ld.bfd --version GNU ld version 2.36.1-8.fc35 Copyright (C) 2021 Free Software Foundation, Inc. This program is free software; you may redistribute it under the terms of the GNU General Public License version 3 or (at your option) a later version. This program has absolutely no warranty. <mock-chroot> sh-5.1# cat main.c void hello(); int main(int argc, char **argv) { hello(); return 0; } <mock-chroot> sh-5.1# cat hello.c #include <stdio.h> void hello() { printf("Hello World\n"); } <mock-chroot> sh-5.1# clang -flto -o main main.c hello.c /usr/bin/ld: BFD version 2.36.1-8.fc35 internal error, aborting at elfcode.h:224 in bfd_elf32_swap_symbol_out /usr/bin/ld: Please report this bug. clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
Created attachment 1785606 [details] Proposed patch Hi Tom, Please could you try this patch. It has been made against the latest GNU Binutils sources, so it may not apply if you are using some other version, but if you let me know which version that is I can generate a new patch. The patch will probably not solve the problem, but it should prevent the illegal memory access, and it might even generate a helpful error message. I still think that the LTO plugin is to blame, but I am not sure where a full fix should be applied. Cheers Nick
Any update on this pretty plz?
@nickc The Fedora arm test machines are back online now: https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers
Also, I just did a build of binutils-2.37 on the test machine and the segfault is gone, but I can still reproduce it in the emulated mock chroot. I wonder if maybe one of the compile flags we use in Fedora is causing some kind of mis-compile.
(In reply to Tom Stellard from comment #18) Hi Tom, > Also, I just did a build of binutils-2.37 on the test machine and the > segfault is gone, but I can still reproduce it in the emulated mock chroot. Was that a Fedora 34 mock chroot ? F34 binutils is based on 2.35.2, so possibly this is a bug that has been fixed between 2.35.2 and 2.37. > I wonder if maybe one of the compile flags we use in Fedora is causing some > kind of mis-compile. If you configured the 2.37 binutils without the "--enable-plugins" option then this might explain why it worked. Cheers Nick
(In reply to Nick Clifton from comment #19) > (In reply to Tom Stellard from comment #18) > Hi Tom, > > > Also, I just did a build of binutils-2.37 on the test machine and the > > segfault is gone, but I can still reproduce it in the emulated mock chroot. > > Was that a Fedora 34 mock chroot ? F34 binutils is based on 2.35.2, so > possibly this is a bug that has been fixed between 2.35.2 and 2.37. > It was a rawhide mock chroot. > > I wonder if maybe one of the compile flags we use in Fedora is causing some > > kind of mis-compile. > > If you configured the 2.37 binutils without the "--enable-plugins" option > then this might explain why it worked. > I just checked and I did configure with --enable-plugins. > Cheers > Nick
On my local builds of binutils, if I drop the -flto=auto flag when building binutils, then the problem goes away. I'm going to do a scratch build of binutils to confirm that this works in the RPM build too.
(In reply to Tom Stellard from comment #21) > On my local builds of binutils, if I drop the -flto=auto flag when building > binutils, then the problem goes away. This is worrying as I thought that building with LTO enabled was getting to be reliable these days. If you can confirm the workaround however then I will update the binutils.spec file to disable LTO when building the binutils for the ARM architecture.
I can confirm that this pull request fixes the issue: https://src.fedoraproject.org/rpms/binutils/pull-request/29
In which case how about we CLOSE this BZ ? :-)
(In reply to Nick Clifton from comment #24) > In which case how about we CLOSE this BZ ? :-) Sure, unless you want to link it to a Bodhi update.
FEDORA-2021-566b3f3e1b has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-566b3f3e1b
FEDORA-2021-74394f7f1a has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-74394f7f1a
Fixed in binutils-2.35.2-6.fc34 Fixed in binutils-2.37-10.fc35 Fixed in binutils-2.37-12.fc36
FEDORA-2021-566b3f3e1b has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-566b3f3e1b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-566b3f3e1b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-74394f7f1a has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-74394f7f1a` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-74394f7f1a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-74394f7f1a has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2021-566b3f3e1b has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.