Bug 2060755
| Summary: | Firefox: GCC 12 linking error | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Martin Stransky <stransky> |
| Component: | firefox | Assignee: | Gecko Maintainer <gecko-bugs-nobody> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 36 | CC: | awilliam, bcotton, bruno, erack, gecko-bugs-nobody, jakub, jhorak, kai-engert-fedora, klaas, nickc, pjasicek, renault, rhughes, rstrode, sandmann, Simon.Gerhards |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | firefox-98.0-3.fc36 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-09 20:09:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2057193 | ||
|
Description
Martin Stransky
2022-03-04 08:25:19 UTC
Jakub, Is there any way how to debug it? The command line is: /usr/bin/g++ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wno-invalid-offsetof -Wc++2a-compat -Wduplicated-cond -Wimplicit-fallthrough -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wno-error=deprecated-copy -Wformat -Wformat-overflow=2 -Wno-psabi -fno-sized-deallocation -fno-aligned-new -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fpermissive -fPIC -Wl,-z,relro -Wl,-z,now -DNSS_PKCS11_3_0_STRICT -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -O3 -fomit-frame-pointer -funwind-tables -DNS_FREE_PERMANENT_DATA=1 -fprofile-generate -shared -Wl,-z,defs -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -fstack-protector-strong -Wl,-rpath-link,/builddir/build/BUILD/firefox-98.0/objdir/instrumented/dist/bin -Wl,-rpath-link,/usr/lib -fprofile-generate /builddir/build/BUILD/firefox-98.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols Adding -lstdc++ does not help. Thanks. Clang claims: 15:22.17 ld.lld: error: undefined hidden symbol: std::__glibcxx_assert_fail(char const*, int, char const*, char const*) 15:22.17 >>> referenced by stl_vector.h:1229 (/usr/include/c++/12/bits/stl_vector.h:1229) 15:22.17 >>> /home/komat/rpmbuild/BUILD/firefox-98.0/objdir/toolkit/library/build/../../../security/sandbox/linux/broker/Unified_cpp_linux_broker0.o:(mozilla::SandboxBrokerPolicyFactory::InitContentPolicy()) 15:22.17 >>> referenced by basic_string.h:1221 (/usr/include/c++/12/bits/basic_string.h:1221) 15:22.17 >>> /home/komat/rpmbuild/BUILD/firefox-98.0/objdir/toolkit/library/build/../../../security/sandbox/linux/launch/Unified_cpp_linux_launch0.o:(std::__cxx11::to_string(int)) 15:22.17 >>> referenced by stl_vector.h:1229 (/usr/include/c++/12/bits/stl_vector.h:1229) 15:22.17 >>> /home/komat/rpmbuild/BUILD/firefox-98.0/objdir/toolkit/library/build/../../../security/sandbox/linux/launch/Unified_cpp_linux_launch0.o:(std::pair<int, int>& std::vector<std::pair<int, int>, std::allocator<std::pair<int, int> > >::emplace_back<std::pair<int, int> >(std::pair<int, int>&&) (.isra.0)) 15:22.17 >>> referenced 2221 more times 15:22.17 >>> did you mean: std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@GLIBCXX_3.4.30 Upstream bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1758005 Nick, any ideas what is going on?
ld command line is:
ld -plugin /usr/libexec/gcc/x86_64-redhat-linux/12/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/x86_64-redhat-linux/12/lto-wrapper -plugin-opt=-fresolution=/tmp/ccCd8ljS.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -shared -o libxul.so /usr/lib/gcc/x86_64-redhat-linux/12/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/12/crtbeginS.o -L/usr/lib64/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/12 -L/usr/lib/gcc/x86_64-redhat-linux/12/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/12/../../.. -z relro -z now -z defs -h libxul.so /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/libxul_so.list -lpthread -z noexecstack -z text -z relro -z nocopyreloc -Bsymbolic-functions --build-id=sha1 -rpath-link /builddir/build/BUILD/firefox-98.0/objdir/instrumented/dist/bin -rpath-link /usr/lib ../../../js/src/build/libjs_static.a /builddir/build/BUILD/firefox-98.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/libgkrust.a ../../../security/sandbox/linux/libmozsandbox.so ../../../config/external/lgpllibs/liblgpllibs.so ../../../config/external/sqlite/libmozsqlite3.so ../../../widget/gtk/mozgtk/libmozgtk.so ../../../widget/gtk/mozwayland/libmozwayland.so --version-script symverscript -lasound -lrt -ldl -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrandr -lXrender -lXtst -lpthread -lffi -lplds4 -lplc4 -lnspr4 -lz -lssl3 -lsmime3 -lnss3 -lnssutil3 -lfreetype -lfontconfig -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -ldbus-glib-1 -ldbus-1 -lxcb-shm -lX11-xcb -lxcb -lXcursor -lXi -lstdc++ -lm -lc -lgcov -lgcc_s -lpthread -lc -lgcc_s /usr/lib/gcc/x86_64-redhat-linux/12/crtendS.o /usr/lib/gcc/x86_64-redhat-linux/12/../../../../lib64/crtn.o
where libxul_so.list includes among many others:
INPUT("../../../security/sandbox/linux/broker/Unified_cpp_linux_broker0.o")
line.
readelf -Wa ../../../security/sandbox/linux/broker/Unified_cpp_linux_broker0.o | grep _ZSt21__glibcxx_assert_failPKciS0_S0_
00000000000000e4 000009fa00000004 R_X86_64_PLT32 0000000000000000 _ZSt21__glibcxx_assert_failPKciS0_S0_ - 4
2554: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZSt21__glibcxx_assert_failPKciS0_S0_
readelf -Wa /usr/lib/gcc/x86_64-redhat-linux/12/libstdc++.so | grep _ZSt21__glibcxx_assert_failPKciS0_S0_
000000000022e9b8 000010f900000007 R_X86_64_JUMP_SLOT 00000000000d7cf0 _ZSt21__glibcxx_assert_failPKciS0_S0_@@GLIBCXX_3.4.30 + 0
4345: 00000000000d7cf0 53 FUNC GLOBAL DEFAULT 14 _ZSt21__glibcxx_assert_failPKciS0_S0_@@GLIBCXX_3.4.30
When I add -M option, it prints
LOAD /usr/lib/gcc/x86_64-redhat-linux/12/libstdc++.so
among many other things, so why isn't the _ZSt21__glibcxx_assert_failPKciS0_S0_ UND symbol satisfied by the libstdc++.so symbol?
Ah, the bug might be that while in most object files that symbol is GLOBAL DEFAULT UND, in a few files it is HIDDEN: File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../ipc/glue/Unified_cpp_ipc_glue1.o 13640: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../gfx/layers/Unified_cpp_gfx_layers1.o 8600: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../dom/base/nsFrameMessageManager.o 4317: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../dom/base/Unified_cpp_dom_base0.o 11213: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../gfx/angle/targets/translator/ValidateOutputs.o 577: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../gfx/angle/targets/angle_common/PackedEnums.o 417: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../dom/media/platforms/agnostic/bytestreams/Unified_cpp_bytestreams0.o 968: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../dom/quota/Unified_cpp_dom_quota0.o 18522: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../dom/localstorage/Unified_cpp_dom_localstorage0.o 13796: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../gfx/skia/Unified_cpp_gfx_skia15.o 11777: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../third_party/libwebrtc/api/video/video_frame_metadata_gn/Unified_cpp_eo_frame_metadata_gn0.o 129: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../third_party/libwebrtc/modules/audio_coding/neteq_gn/audio_vector.o 522: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../third_party/libwebrtc/modules/rtp_rtcp/rtp_rtcp_gn/video_rtp_depacketizer_vp9.o 225: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../../third_party/libwebrtc/modules/video_coding/webrtc_multiplex_gn/Unified_cpp_webrtc_multiplex_gn0.o 2096: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../components/telemetry/TelemetryHistogram.o 3811: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ File: /builddir/build/BUILD/firefox-98.0/objdir/instrumented/toolkit/library/build/../../xre/Unified_cpp_toolkit_xre0.o 2013: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND _ZSt21__glibcxx_assert_failPKciS0_S0_ and perhaps if at least one of them is hidden it requires that the symbol is defined in the libxul.so library rather than elsewhere. So, the issue is that e.g.
/builddir/build/BUILD/firefox-98.0/gfx/angle/checkout/src/common/PackedEnums.cpp
has (reduced for brevity from preprocessed source):
# 0 "/builddir/build/BUILD/firefox-98.0/gfx/angle/checkout/src/common/PackedEnums.cpp"
# 1 "/builddir/build/BUILD/firefox-98.0/objdir/instrumented/gfx/angle/targets/angle_common//"
# 1 "/builddir/build/BUILD/firefox-98.0/config/gcc_hidden.h" 1
#pragma GCC visibility push(hidden)
# 33 "/usr/include/c++/12/cstdint" 3
# 2781 "/usr/include/c++/12/x86_64-redhat-linux/bits/c++config.h" 3
namespace std
{
extern "C++" __attribute__ ((__noreturn__))
void
__glibcxx_assert_fail(const char* __file, int __line,
const char* __function, const char* __condition)
noexcept;
}
and so __glibcxx_assert_fail will be hidden.
So perhaps:
--- libstdc++-v3/include/bits/c++config.jj 2022-02-14 16:52:26.193973005 +0100
+++ libstdc++-v3/include/bits/c++config 2022-03-04 18:20:26.954581207 +0100
@@ -550,7 +550,7 @@ namespace std
#if defined(_GLIBCXX_ASSERTIONS) \
|| defined(_GLIBCXX_PARALLEL) || defined(_GLIBCXX_PARALLEL_ASSERTIONS)
# ifdef _GLIBCXX_VERBOSE_ASSERT
-namespace std
+namespace std _GLIBCXX_VISIBILITY(default)
{
// Avoid the use of assert, because we're trying to keep the <cassert>
// include out of the mix.
Though, there are various other extern declarations in c++config, so maybe some of them need it too.
Can we try that, or similar? This is quite critical as it is preventing Firefox from building, and that is blocking the F36 Beta release. We have the first go/no-go for F36 Beta on Thursday, which means that we really need blocking issues resolved today or latest tomorrow or else we will miss that first release target. I'm building gcc with that for almost 31 hours already: https://koji.fedoraproject.org/koji/taskinfo?taskID=83718063 https://koji.fedoraproject.org/koji/taskinfo?taskID=83718108 But something wrong is happening on i686 in both of those builds. Dunno if we've just run into limits of LTO building of gcc on 32-bit arches (on armv7hl we are disabling it already from earlier) or what else is happening. Builds from less than fortnight ago went just fine. Note, just for testing whether it fixes the firefox problems e.g. the x86_64 or other arch binary rpms (except for i686) from those builds can be grabbed and used in local mock. A short update from the gcc side. The i686 problems apparently are due to the Ada compiler binaries hangs at some point, I suspect they segfault or get other fatal signal before the relocations are fully resolved and in the handler ada__exceptions__exception_propagation__allocate_occurrenceXn call results in NULL in the plt slot and is caught again and like this forever. I can reproduce it in koji and local mock (and it seems to happen regardless of whether it is a GCC lto bootstrap or non-lto, pgo or non-pgo), but strangely I can't reproduce it if I run the same command by hand in mock shell afterwards, so it is kind of a nightmare to debug. So, I've instead bisected it which gcc change triggers it and it seems it is the https://gcc.gnu.org/r12-7472 change. I'll try to add instrumentation to that function to see in which TUs/functions it changes behavior. https://koji.fedoraproject.org/koji/taskinfo?taskID=83832384 https://koji.fedoraproject.org/koji/taskinfo?taskID=83832394 new builds with the i686 hangs hopefully fixed. But ETA 24h or more if all goes well. (In reply to Jakub Jelinek from comment #4) [Just clearing the needinfo...] You have it right - the symbol is hidden so it cannot be used to resolve an external reference. gcc-12.0.1-0.12.fc37 in rawhide should fix this, f36 version still building... https://koji.fedoraproject.org/koji/buildinfo?buildID=1929012 seems to have worked for firefox on rawhide FEDORA-2022-9fb66de66d has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-9fb66de66d FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-42ea499a7d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-42ea499a7d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report. |