It appears that Firefox cannot be built with this particular version of gcc. The last successful build for F42 in koji is using release 0.9, which does not include some of the patches that 0.11 has. I am still confirming all this by running a build in copr of the official Firefox build that exists in F42 (137.0-2): https://copr.fedorainfracloud.org/coprs/bojan/FF/build/8876055/. This will take a few hours... I ran a diff between source tarballs of 137.0 and 137.0.1 and it seems that only one C++ file was changed and one does did not seem to be involved in this compilation failure of skia. Of course, I could be totally wrong on all this, ergo I'm including @stransky on CC of this bug. But, I thought I'd raise the possibility of breakage, just in case. Reproducible: Always Steps to Reproduce: 1. Build Firefox in koji/copr with gcc-15.0.1-0.11.fc42.x86_64 Actual Results: Stuff like this: 92:28.89 /usr/bin/g++ -o Unified_cpp_gfx_layers11.o -c -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/dist/stl_wrappers -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/dist/system_wrappers -include /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstrict-flex-arrays=1 -DNDEBUG=1 -DTRIMMED=1 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DMOZ_APP_VERSION=137.0.1 -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DMOZ_SUPPORT_LEAKCHECKING -DSTATIC_EXPORTABLE_JS_API -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/layers -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/gfx/layers -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/ipc/ipdl/_ipdlheaders -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/ipc/chromium/src -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/docshell/base -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/dom/canvas -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/cairo/cairo/src -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/layout/base -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/layout/generic -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/media/libyuv/libyuv/include -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/dist/include -I/usr/include/nspr4 -I/usr/include/nss3 -I/usr/include/nspr4 -I/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/dist/include/nss -DWITH_GZFILEOP -I/usr/include/pixman-1 -DMOZILLA_CLIENT -include /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/mozilla-config.h -fno-rtti -pthread -fno-sized-deallocation -fno-aligned-new -ffunction-sections -fdata-sections -fno-math-errno -fno-exceptions -pipe -fPIC -O2 -g1 -grecord-gcc-switches -pipe -Wno-complain-wrong-lang -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpermissive -DNSS_PKCS11_3_0_STRICT -O3 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wno-invalid-offsetof -Wcomma-subscript -Wvolatile -Wno-deprecated-enum-enum-conversion -Wduplicated-cond -Wimplicit-fallthrough -Wlogical-op -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wformat -Wformat-overflow=2 -Wno-psabi -Wno-error=builtin-macro-redefined -I/usr/include/gtk-3.0/unix-print -I/usr/include/gtk-3.0 -I/usr/include/pango-1.0 -I/usr/include/cloudproviders -I/usr/include/at-spi2-atk/2.0 -I/usr/include/at-spi-2.0 -I/usr/lib64/pkgconfig/../../include/dbus-1.0 -I/usr/lib64/pkgconfig/../../lib64/dbus-1.0/include -I/usr/include/fribidi -I/usr/include/harfbuzz -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/libxml2 -I/usr/include/freetype2 -I/usr/include/pixman-1 -I/usr/include/gdk-pixbuf-2.0 -I/usr/include/libpng16 -I/usr/include/webp -I/usr/include/gio-unix-2.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -DWITH_GZFILEOP -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/sysprof-6 -pthread -Werror=switch -Wno-maybe-uninitialized -fno-strict-aliasing -ffp-contract=off -DNS_FREE_PERMANENT_DATA=1 -fprofile-generate -MD -MP -MF .deps/Unified_cpp_gfx_layers11.o.pp Unified_cpp_gfx_layers11.cpp 92:30.05 In file included from /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/core/SkOpts.cpp:16: 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h: In function ‘void sse2::trace_scope(SkRasterPipelineStage*, size_t, size_t, std::byte*, F, F, F, F, F, F, F, F)’: 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1603:31: error: cannot tail-call: other reasons 92:30.06 1603 | MUSTTAIL return fn(program, dx,dy,base, r,g,b,a, dr,dg,db,da); \ 92:30.06 | ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1618:5: note: in expansion of macro ‘DECLARE_STAGE’ 92:30.06 1618 | DECLARE_STAGE(name, arg, void, ++program, /*no offset*/, SKRP_MUSTTAIL) 92:30.06 | ^~~~~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:4050:1: note: in expansion of macro ‘STAGE_TAIL’ 92:30.06 4050 | STAGE_TAIL(trace_scope, SkRasterPipeline_TraceScopeCtx* ctx) { 92:30.06 | ^~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h: In function ‘void sse2::trace_exit(SkRasterPipelineStage*, size_t, size_t, std::byte*, F, F, F, F, F, F, F, F)’: 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1603:31: error: cannot tail-call: other reasons 92:30.06 1603 | MUSTTAIL return fn(program, dx,dy,base, r,g,b,a, dr,dg,db,da); \ 92:30.06 | ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1618:5: note: in expansion of macro ‘DECLARE_STAGE’ 92:30.06 1618 | DECLARE_STAGE(name, arg, void, ++program, /*no offset*/, SKRP_MUSTTAIL) 92:30.06 | ^~~~~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:4043:1: note: in expansion of macro ‘STAGE_TAIL’ 92:30.06 4043 | STAGE_TAIL(trace_exit, SkRasterPipeline_TraceFuncCtx* ctx) { 92:30.06 | ^~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h: In function ‘void sse2::trace_enter(SkRasterPipelineStage*, size_t, size_t, std::byte*, F, F, F, F, F, F, F, F)’: 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1603:31: error: cannot tail-call: other reasons 92:30.06 1603 | MUSTTAIL return fn(program, dx,dy,base, r,g,b,a, dr,dg,db,da); \ 92:30.06 | ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1618:5: note: in expansion of macro ‘DECLARE_STAGE’ 92:30.06 1618 | DECLARE_STAGE(name, arg, void, ++program, /*no offset*/, SKRP_MUSTTAIL) 92:30.06 | ^~~~~~~~~~~~~ 92:30.06 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:4036:1: note: in expansion of macro ‘STAGE_TAIL’ 92:30.06 4036 | STAGE_TAIL(trace_enter, SkRasterPipeline_TraceFuncCtx* ctx) { 92:30.06 | ^~~~~~~~~~ 92:30.07 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h: In function ‘void sse2::trace_line(SkRasterPipelineStage*, size_t, size_t, std::byte*, F, F, F, F, F, F, F, F)’: 92:30.07 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1603:31: error: cannot tail-call: other reasons 92:30.07 1603 | MUSTTAIL return fn(program, dx,dy,base, r,g,b,a, dr,dg,db,da); \ 92:30.07 | ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 92:30.07 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:1618:5: note: in expansion of macro ‘DECLARE_STAGE’ 92:30.07 1618 | DECLARE_STAGE(name, arg, void, ++program, /*no offset*/, SKRP_MUSTTAIL) 92:30.07 | ^~~~~~~~~~~~~ 92:30.07 /builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/gfx/skia/skia/src/opts/SkRasterPipeline_opts.h:4029:1: note: in expansion of macro ‘STAGE_TAIL’ 92:30.07 4029 | STAGE_TAIL(trace_line, SkRasterPipeline_TraceLineCtx* ctx) { 92:30.07 | ^~~~~~~~~~ 92:30.65 gmake[4]: *** [/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/config/rules.mk:676: SkOpts.o] Error 1 92:30.66 gmake[4]: Leaving directory '/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/objdir/instrumented/gfx/skia' 92:30.66 gmake[3]: *** [/builddir/build/BUILD/firefox-137.0.1-build/firefox-137.0.1/config/recurse.mk:72: gfx/skia/target-objects] Error 2 92:30.66 gfx/layers/Unified_cpp_gfx_layers3.o 92:30.66 gmake[3]: *** Waiting for unfinished jobs.... Expected Results: Used to work with release 0.9
Yep, 137.0-2 (from koji built src) bombed out, just like 137.0.1.
Do you have preprocessed source? I've fixed various musttail related issues since the last gcc snapshot in rawhide: https://gcc.gnu.org/PR119376 https://gcc.gnu.org/PR119491 https://gcc.gnu.org/PR119493 https://gcc.gnu.org/PR119535 https://gcc.gnu.org/PR119613 https://gcc.gnu.org/PR119614 https://gcc.gnu.org/PR119616 https://gcc.gnu.org/PR119618 One sub-issue in https://gcc.gnu.org/PR119614 (only with LTO) remains to be worked on tomorrow.
I did those builds in copr and the only artefacts that are left are logs. However, if you know of a way of preserving this code while building the rpm in mock, let me know and I can try on my machine. In any event, thanks for working on this and if you produce any builds of gcc I can test with, let me know and I'll kick off rebuilds in copr again. For that, the builds have to land in testing.
https://koji.fedoraproject.org/koji/taskinfo?taskID=131362476 https://koji.fedoraproject.org/koji/taskinfo?taskID=131362490 but when exactly it will finish building is unknown.
Thanks! Once available, I'll attempt the build again in copr and report back.
Created a build override and trying again with 0.12 now: https://copr.fedorainfracloud.org/coprs/bojan/FF/build/8889635/
Thank you for the quick fix! Build of 137.0.1 successful in copr with 0.12: https://copr.fedorainfracloud.org/coprs/bojan/FF/build/8889635/ PS. Firefox built with this gcc works. Writing this in it.
Looks like Mozilla is readying 137.0.2 for a release, which will need to be built. Any chance of getting this gcc update into F42?
@stransky, because I wanted to build 137.0.2 in copr (https://copr.fedorainfracloud.org/coprs/bojan/FF/build/8904743/, running as of this writing), there is still an active build override for F42: https://bodhi.fedoraproject.org/overrides/gcc-15.0.1-0.12.fc42. Just FYI.
Judging by build logs of Firefox in koji, even with 0.12, the builds are still failing on s390x.
@jakub, is there something in 0.13 that may help with Firefox builds on s390x? Do you think it's worth trying? PS. The logs seem to be suggesting that those MUSTTAIL calls are still not being understood on that arch.
In the s390x case, what protobuf is trying to do just doesn't work. It boils down to: typedef unsigned char uint8_t; typedef unsigned short uint16_t; typedef unsigned long long uint64_t; struct TcFieldData { constexpr TcFieldData() : data(0) {} constexpr TcFieldData(uint16_t coded_tag, uint8_t hasbit_idx, uint8_t aux_idx, uint16_t offset) : data(uint64_t{offset} << 48 | uint64_t{aux_idx} << 24 | uint64_t{hasbit_idx} << 16 | uint64_t{coded_tag}) {} uint64_t data; }; const char *foo (void *, void *, void *, void *, uint64_t, TcFieldData); [[gnu::noinline]] const char *bar (void *, void *, void *, void *, uint64_t, TcFieldData) { return nullptr; } const char * baz (void *a, void *b, void *c, void *d, uint64_t e, TcFieldData f) { [[clang::musttail]] return bar (a, b, c, d, e, f); } const char * qux (void *a, void *b, void *c, void *d, uint64_t e, TcFieldData f) { [[clang::musttail]] return foo (a, b, c, d, e, f); } This fails to compile with GCC 15/16: /tmp/0.C: In function ‘const char* baz(void*, void*, void*, void*, uint64_t, TcFieldData)’: /tmp/0.C:18:34: error: cannot tail-call: target is not able to optimize the call into a sibling call 18 | [[clang::musttail]] return bar (a, b, c, d, e, f); | ~~~~^~~~~~~~~~~~~~~~~~ /tmp/0.C: In function ‘const char* qux(void*, void*, void*, void*, uint64_t, TcFieldData)’: /tmp/0.C:24:34: error: cannot tail-call: target is not able to optimize the call into a sibling call 24 | [[clang::musttail]] return foo (a, b, c, d, e, f); | ~~~~^~~~~~~~~~~~~~~~~~ While clang++ apparently compiles that fine, it actually doesn't tail call it, performs normal call without diagnosing it (which is IMHO a severe bug). See https://godbolt.org/z/95c1dzs3P where clang emits: _Z3bazPvS_S_S_y11TcFieldData: stmg %r14, %r15, 112(%r15) aghi %r15, -168 brasl %r14, _Z3barPvS_S_S_y11TcFieldData@PLT lmg %r14, %r15, 280(%r15) br %r14 _Z3quxPvS_S_S_y11TcFieldData: stmg %r14, %r15, 112(%r15) aghi %r15, -168 lg %r0, 328(%r15) stg %r0, 160(%r15) brasl %r14, _Z3fooPvS_S_S_y11TcFieldData@PLT lmg %r14, %r15, 280(%r15) br %r14 The reason why this can't be tail-called on s390 is an ABI deficiency, as described in the GCC s390 backend: /* Register 6 on s390 is available as an argument register but unfortunately "caller saved". This makes functions needing this register for arguments not suitable for sibcalls. */ So, in short, given normal integral/pointer arguments, one can only tailcall calls with up to 4 arguments, not more on s390. To be honest, I really don't know why this couldn't be handled in this particular case where the 5th argument is just passed through from caller to the musttail callee, because in that case nothing should change the caller saved register. I haven't checked the protobuf cases whether the 5th argument is there always pass through and not sometimes modified (in that case really the ABI doesn't allow tail calling it. Anyway, I'd suggest to patch protobuf, something like --- toolkit/components/protobuf/src/google/protobuf/port_def.inc 2025-04-14 14:07:32.000000000 +0200 +++ toolkit/components/protobuf/src/google/protobuf/port_def.inc 2025-04-19 18:43:29.695583024 +0200 @@ -257,12 +257,14 @@ #if __has_cpp_attribute(clang::musttail) && !defined(__arm__) && \ !defined(_ARCH_PPC) && !defined(__wasm__) && \ !(defined(_MSC_VER) && defined(_M_IX86)) && \ - !(defined(__NDK_MAJOR__) && __NDK_MAJOR <= 24) + !(defined(__NDK_MAJOR__) && __NDK_MAJOR <= 24) && \ + !defined(__s390__) # ifndef PROTO2_OPENSOURCE // Compilation fails on ARM32: b/195943306 // Compilation fails on powerpc64le: b/187985113 // Compilation fails on X86 Windows: // https://github.com/llvm/llvm-project/issues/53271 +// Compilation fails on S390 for calls with more than 4 arguments # endif #define PROTOBUF_MUSTTAIL [[clang::musttail]] #define PROTOBUF_TAILCALL true (untested). Because at least clang 13-20 including current trunk and gcc 15 and current trunk don't really support this.
I've filed it in GCC bugzilla, but just as an enhancement request for the passthrough case.
Thanks @jakub! I will move the bug to protobuf component to get feedback from maintainers.
If I'm reading Firefox spec file correctly, bundled copy of protobuf is being used there, so your patch likely has to be applied directly in Firefox builds as well.
@jakub, the patch works! Thank you!
FEDORA-2025-d92ef6c3bc (firefox-137.0.2-3.fc43) has been submitted as an update to Fedora 43. https://bodhi.fedoraproject.org/updates/FEDORA-2025-d92ef6c3bc
FEDORA-2025-d92ef6c3bc (firefox-137.0.2-3.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
@jakub, could we get gcc-15.0.1-0.14 into F42, so that builds of Firefox work there without build overrides?
Can you use a build override for now? I expect to build gcc-15.1.0-1.fc42 (i.e. GCC 15.1 final release) on Friday or so, so it wouldn't make sense to push an errata only to have another one pushed in 5 days from that.
ACK. Thanks!
FYI, as I wasn't aware of this bug I pushed https://github.com/protocolbuffers/protobuf/commit/a2859cc2ce25711613002104022186c0c37d9f1f into protobuf upstream