I can see it too. Looks like caused by GCC 16 update in Rawhide.

This is PGO+LTO build. I'll try a plain one.

> All builds of Firefox that have been done recently with the current Rawhide build chain:

Is the issue reported in mzbz#1999625 (https://bugzilla.mozilla.org/show_bug.cgi?id=1999625) fixed?
How?

Tested a build without PGO+LTO and it works.
mzbz#1999625 is "fixed" by local patch.

Okay, local PGO+LTO build crashes too. I'll try non-unified build. I can provide the affected nsStandardURL.cpp file compiled if neccessary.

Have you tried just LTO without PGO or PGO without LTO?
In any case, which library or binary is this in (supposedly whatever contains nsStandardURL.cpp)?
If it is LTO, just preprocessed nsStandardURL.cpp will not be enough though.

Looking up upstream, we have some ICE on firefox reported - https://gcc.gnu.org/PR123229 but am not aware of a firefox related miscompilation.

Anyway, depending on when it can be reproduced (LTO+PGO only, or even with just PGO or even with just LTO) and on which library or binary, ideally it should be reduced to as few translation units as possible.
If reproduceable just with PGO and narrowed to a single TU, that TU + its corresponding *.gcda file is what is needed for analysis.
If LTO is needed, trying to recompile separately all the *.o files linked into that library or binary with -fno-lto and then bisecting between -flto and -fno-lto compiled objects during link could narrow it down
to smallest subset of -flto built TUs, if there are say 1-10, those can be preprocessed with older compiler version and then bisected among gcc revisions.
I guess I can do that but at least the answer to #c5 would help me with that (+ how to reproduce as quickly as possible from the mock build idealy on F42 desktop without affecting normal firefox profile there).

Created attachment 2123836 [details]
libxul_so.list.bad7

Created attachment 2123838 [details]
libxul_so.list.good1

Ok, I've managed to reproduce this.
Did two builds of firefox, one with LTO+PGO disabled through changing
%global build_with_pgo    1
to
%global build_with_pgo    0
which doesn't reproduce the crash and one with LTO+PGO enabled which does reproduce the crash, confirmed the non-LTO/PGO build with libxul.so from the LTO+PGO build crashes and then went onto bisection during libxul.so linking.
I've copied over the non-LTO/PGO objdir into objdir.good next to objdir and copied objdir from LTO+PGO build to objdir.bad.
And then have been linking
/usr/bin/g++ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstrict-flex-arrays=1 -fno-rtti -pthread -fno-sized-deallocation -fno-aligned-new -ffunction-sections -fdata-sections -fno-math-errno -fno-exceptions -pipe -fPIC -O2 -g1 -grecord-gcc-switches -pipe -Wno-complain-wrong-lang -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fpermissive -DNSS_PKCS11_3_0_STRICT -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o ../../../dist/bin/libxul.so -Wl,@/builddir/build/BUILD/firefox-147.0.1-build/firefox-147.0.1/objdir/toolkit/library/build/libxul_so.list   -flto=32 -flifetime-dse=1 -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -Wl,--build-id=sha1 -Wl,--no-keep-memory -Wl,--build-id=sha1 -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -Wl,-z,pack-relative-relocs -fstack-protector-strong -Wl,-rpath-link,/builddir/build/BUILD/firefox-147.0.1-build/firefox-147.0.1/objdir/dist/bin -fprofile-use ../../../../objdir.good/js/src/build/libjs_static.a ../../../../objdir.good/build/pure_virtual/libpure_virtual.a ../../../../objdir.good/x86_64-unknown-linux-gnu/release/libgkrust.a ../../../../objdir.good/dist/bin/libmozsandbox.so ../../../../objdir.good/dist/bin/libgkcodecs.so ../../../../objdir.good/dist/bin/liblgpllibs.so ../../../../objdir.good/dist/bin/libmozsqlite3.so ../../../../objdir.good/dist/bin/libmozgtk.so ../../../../objdir.good/dist/bin/libmozwayland.so -Wl,--version-script,libxul.so.symbols  -lresolv -ldl -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrandr -lXrender -ldrm -lpipewire-0.3 -lasound -lgbm -lpthread -lc -lffi -lplds4 -lplc4 -lnspr4 -lz -lm -lssl3 -lsmime3 -lnss3 -lnssutil3 -lfreetype -lfontconfig -lgtk-3 -lgdk-3 -lpangocairo-1.0 -lpango-1.0 -lharfbuzz -latk-1.0 -lcairo-gobject -lcairo -lgdk_pixbuf-2.0 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -lrt -ljpeg -lwebp -lwebpdemux -levent -lvpx -lpixman-1 -L/usr/lib64/pkgconfig/../../lib64 -ldbus-1 -lxcb-shm -lX11-xcb -lxcb -lXcursor -lXi
in /builddir/build/BUILD/firefox-147.0.1-build/firefox-147.0.1/objdir/toolkit/library/build
If libxul_so.list is the good1 one, then it doesn't crash, if it is the bad7 one, then it crashes, the difference is
-../../../../objdir.good/toolkit/xre/nsAppRunner.o
+../../../../objdir.bad/toolkit/xre/nsAppRunner.o
So, if something is miscompiled, it is likely something from that file or related to that.
Note, there are a few other *.o files coming from the LTO+PGO build, because for some strange reason those are completely missing from the non-LTO/PGO build:
../../../ipc/glue/test/utility_process_xpcom/Unified_cpp_tility_process_xpcom0.o
../../../js/xpconnect/tests/components/native/Unified_cpp_components_native0.o
../../../dom/media/test/rdd_process_xpcom/Unified_cpp_rdd_process_xpcom0.o
../../../third_party/opentelemetry-cpp/exporters/otlp/Unified_cpp_exporters_otlp0.o
../../../third_party/opentelemetry-cpp/third_party/opentelemetry-proto/trace_service.pb.o
../../../third_party/opentelemetry-cpp/third_party/opentelemetry-proto/common.pb.o
../../../third_party/opentelemetry-cpp/third_party/opentelemetry-proto/resource.pb.o
../../../third_party/opentelemetry-cpp/third_party/opentelemetry-proto/trace.pb.o
../../../third_party/opentelemetry-cpp/sdk/src/common/Unified_cpp_sdk_src_common0.o
../../../third_party/opentelemetry-cpp/sdk/src/resource/Unified_cpp_sdk_src_resource0.o
../../../third_party/opentelemetry-cpp/sdk/src/trace/Unified_cpp_sdk_src_trace0.o
../../../third_party/opentelemetry-cpp/sdk/src/version/version.o
../../../third_party/opentelemetry-cpp/exporters/memory/Unified_cpp_exporters_memory0.o

Ok, rebuilt by hand using commands from build.log the above 13 object files with additional -fno-lto and without -fprofile-use -fprofile-correction and with libxul_so.list the same as the bad7 one it still crashes, so verified in debug info it is about a single TU compiled with -flto, nsAppRunner.o.

Created attachment 2123853 [details]
nsAppRunner.ii.xz

This is just weird.  So, I can reproduce the crash also if I link in nsAppRunner.o from the LTO+PGO build, but compiled without -fprofile-use, or even compiled without -flto, or even compiled with -fno-lto -O0.
Attaching the preprocessed source.
But it works fine if I link nsAppRunner.o from the non-LGO/PGO build.
So, at this point this is not about how it is compiled, but about what is compiled.
So, I wonder if nsAppRunner.cc somehow doesn't have preprocessero guarded code or something similar depending on LTO+PGO vs. normal build and it isn't what is crashing.  Or something dependent on configure.

Created attachment 2123856 [details]
nsAppRunner.ii.xz

And here is nsAppRunner.ii from the working (i.e. non-LTO/PGO) build.
There are quite a few differences,
   PHttpConnectionMgrMsgStart,
   PHttpTransactionMsgStart,
   PIPCClientCertsMsgStart,
+  PIPDLUnitTestMsgStart,
   PIdleSchedulerMsgStart,
   PImageBridgeMsgStart,
   PInProcessMsgStart,
@@ -203478,6 +203479,7 @@ enum IPCMessageStart {
   PProxyConfigLookupMsgStart,
   PQuotaMsgStart,
   PQuotaRequestMsgStart,
+  PQuotaTestMsgStart,
   PQuotaUsageRequestMsgStart,
   PRDDMsgStart,
   PRemoteCDMMsgStart,
@@ -203514,8 +203516,39 @@ enum IPCMessageStart {
   PTCPSocketMsgStart,
   PTRRServiceMsgStart,
   PTemporaryIPCBlobMsgStart,
+  PTestAsyncReturnsMsgStart,
+  PTestBasicMsgStart,
+  PTestCancelMsgStart,
+  PTestCrossProcessSemaphoreMsgStart,
+  PTestDataStructuresMsgStart,
+  PTestDataStructuresSubMsgStart,
+  PTestDescendantMsgStart,
+  PTestDescendantSubMsgStart,
+  PTestDescendantSubsubMsgStart,
+  PTestDestroyNestedMsgStart,
+  PTestDestroyNestedSubMsgStart,
+  PTestEndpointOpensMsgStart,
+  PTestEndpointOpensOpenedMsgStart,
+  PTestHangsMsgStart,
+  PTestInduceConnectionErrorMsgStart,
+  PTestJSONMsgStart,
+  PTestJSONHandleMsgStart,
+  PTestManyChildAllocsMsgStart,
+  PTestManyChildAllocsSubMsgStart,
+  PTestManyHandlesMsgStart,
+  PTestMostNestedMsgStart,
+  PTestMultiMgrsMsgStart,
+  PTestMultiMgrsBottomMsgStart,
+  PTestMultiMgrsLeftMsgStart,
+  PTestMultiMgrsRightMsgStart,
+  PTestSelfManageMsgStart,
+  PTestSelfManageRootMsgStart,
   PTestShellMsgStart,
   PTestShellCommandMsgStart,
+  PTestShmemMsgStart,
+  PTestSyncErrorMsgStart,
+  PTestUniquePtrIPCMsgStart,
+  PTestUrgencyMsgStart,
   PTextureMsgStart,
   PTransportProviderMsgStart,
   PUDPSocketMsgStart,
@@ -228403,6 +228436,7 @@ enum ID : uint16_t
   OfflineAudioContext,
   OffscreenCanvas,
   OffscreenCanvasRenderingContext2D,
+  OnlyForUseInConstructor,
   OscillatorNode,
   PageTransitionEvent,
   PaintRequest,
@@ -228682,6 +228716,55 @@ enum ID : uint16_t
   TaskPriorityChangeEvent,
   TaskSignal,
   Tensor,
+  TestAttributesOnTypes,
+  TestCEReactionsInterface,
+  TestCImplementedInterface,
+  TestCImplementedInterface2,
+  TestCallbackDictUnionOverload,
+  TestChildInterface,
+  TestConstructorForFuncInterface,
+  TestConstructorForPrefInterface,
+  TestConstructorForSCInterface,
+  TestCppKeywordNamedMethodsInterface,
+  TestDeprecatedInterface,
+  TestExampleInterface,
+  TestExampleProxyInterface,
+  TestExampleThrowingConstructorInterface,
+  TestExampleWorkerInterface,
+  TestFuncConstructorForDifferentFuncInterface,
+  TestFuncConstructorForInterface,
+  TestHTMLConstructorInterface,
+  TestIndexedAndNamedGetterAndSetterInterface,
+  TestIndexedAndNamedGetterInterface,
+  TestIndexedAndNamedSetterInterface,
+  TestIndexedGetterAndSetterAndNamedGetterInterface,
+  TestIndexedGetterInterface,
+  TestIndexedSetterInterface,
+  TestInterface,
+  TestInterfaceWithPromiseConstructorArg,
+  TestJSImplInterface,
+  TestJSImplInterface2,
+  TestJSImplInterface3,
+  TestJSImplInterface4,
+  TestJSImplInterface5,
+  TestJSImplInterface6,
+  TestJSImplNoInterfaceObject,
+  TestLegacyFactoryFunctionInterface,
+  TestLegacyFactoryFunctionInterface2,
+  TestNamedDeleterInterface,
+  TestNamedDeleterWithRetvalInterface,
+  TestNamedGetterInterface,
+  TestNamedSetterInterface,
+  TestNonWrapperCacheInterface,
+  TestParentInterface,
+  TestPrefChromeOnlySCFuncConstructorForInterface,
+  TestPrefConstructorForDifferentPrefInterface,
+  TestPrefConstructorForInterface,
+  TestRenamedInterface,
+  TestSCConstructorForInterface,
+  TestSecureContextInterface,
+  TestThrowingConstructorInterface,
+  TestWorkerExposedInterface,
   TestingDeprecatedInterface,
   Text,
   TextClause,
@@ -229427,6 +229510,7 @@ enum ID : uint16_t
   OfflineAudioContext,
   OffscreenCanvas,
   OffscreenCanvasRenderingContext2D,
+  OnlyForUseInConstructor,
   OscillatorNode,
   PageTransitionEvent,
   PaintRequest,
@@ -229705,7 +229789,58 @@ enum ID : uint16_t
   TaskPriorityChangeEvent,
   TaskSignal,
   Tensor,
+  TestAttributesOnTypes,
+  TestCEReactionsInterface,
+  TestCImplementedInterface,
+  TestCImplementedInterface2,
+  TestCallbackDictUnionOverload,
+  TestChildInterface,
+  TestConstructorForFuncInterface,
+  TestConstructorForPrefInterface,
+  TestConstructorForSCInterface,
+  TestCppKeywordNamedMethodsInterface,
+  TestDeprecatedInterface,
+  TestExampleInterface,
+  TestExampleProxyInterface,
+  TestExampleThrowingConstructorInterface,
+  TestExampleWorkerInterface,
+  TestFuncConstructorForDifferentFuncInterface,
+  TestFuncConstructorForInterface,
+  TestHTMLConstructorInterface,
+  TestIndexedAndNamedGetterAndSetterInterface,
+  TestIndexedAndNamedGetterInterface,
+  TestIndexedAndNamedSetterInterface,
+  TestIndexedGetterAndSetterAndNamedGetterInterface,
+  TestIndexedGetterInterface,
+  TestIndexedSetterInterface,
+  TestInterface,
+  TestInterfaceWithPromiseConstructorArg,
+  TestJSImplInterface,
+  TestJSImplInterface2,
+  TestJSImplInterface3,
+  TestJSImplInterface4,
+  TestJSImplInterface5,
+  TestJSImplInterface6,
+  TestLegacyFactoryFunctionInterface,
+  TestLegacyFactoryFunctionInterface2,
+  TestNamedDeleterInterface,
+  TestNamedDeleterWithRetvalInterface,
+  TestNamedGetterInterface,
+  TestNamedSetterInterface,
+  TestNamespace,
+  TestNonWrapperCacheInterface,
+  TestParentInterface,
+  TestPrefChromeOnlySCFuncConstructorForInterface,
+  TestPrefConstructorForDifferentPrefInterface,
+  TestPrefConstructorForInterface,
+  TestProtoObjectHackedNamespace,
+  TestRenamedInterface,
+  TestRenamedNamespace,
+  TestSCConstructorForInterface,
+  TestSecureContextInterface,
+  TestThrowingConstructorInterface,
   TestUtils,
+  TestWorkerExposedInterface,
   TestingDeprecatedInterface,
   Text,
   TextClause,
@@ -234737,6 +234872,14 @@ struct PrototypeTraits<prototypes::id::O
   };
 };
 template <>
+struct PrototypeTraits<prototypes::id::OnlyForUseInConstructor>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
 struct PrototypeTraits<prototypes::id::OscillatorNode>
 {
   enum
@@ -236969,6 +237112,398 @@ struct PrototypeTraits<prototypes::id::T
   };
 };
 template <>
+struct PrototypeTraits<prototypes::id::TestAttributesOnTypes>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestCEReactionsInterface>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestCImplementedInterface>
+{
+  enum
+  {
+    Depth = 1
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestCImplementedInterface2>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestCallbackDictUnionOverload>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestChildInterface>
+{
+  enum
+  {
+    Depth = 1
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestConstructorForFuncInterface>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestConstructorForPrefInterface>
+{
+  enum
+  {
+    Depth = 0
+  };
+};
+template <>
+struct PrototypeTraits<prototypes::id::TestConstructorForSCInterface>

etc.

Ok, verified if I compile nsAppRunner.ii from the non-LTO/PGO build in the LTO/PGO build, even with -O0 without -flto or -fprofile-use, it doesn't crash on startup, while when
nsRunner.ii from the LTO/PGO build in that build, even with -O0 without -flto or -fprofile-use, it still crashes.
So yes, this is not about what the compiler does, but what the python generation script emit.  I have no idea what they do though, but when that file is -O0 -fno-lto compiled, I guess it shouldn't be hard to debug by somebody familiar with firefox.

Now, whether what the python scripts generate depends on the compiler from earlier, I have no idea.

Created attachment 2123860 [details]
nsAppRunner.ii.tar.xz

Sorry, I might have uploaded one of the preprocessed sources preprocessed with -O0 vs. one non-O0.
Here is a new set, both preprocessed with -O2.
Still, the bad one (i.e. preprocessed in LTO+PGO build) crashes, the good one (i.e. preprocessed in non-LTO/PGO build, compiled/linked in LTO+PGO build) crashes.
Unless this is all a testing problem that firefox binary has to match the build.

*** Bug 2431726 has been marked as a duplicate of this bug. ***

Thanks, will look at it.

firefox-147.0.1-6.fc44.x86_64.rpm ( https://koji.fedoraproject.org/koji/buildinfo?buildID=2925752 ) still broken!. Thanks.

Some progress, if everything in libxul.so link except netwerk/base/Unified_cpp_netwerk_base{3,4}.o has been recompiled with -fno-lto, it still crashes.
And unfortunately the *.gcda files are needed.

Created attachment 2127463 [details]
rh2431315.tar.xz

The corresponding gcda files and preprocessed sources and commands.

I think the problem is related to the speculative devirtualization in NewStandardURI function, from the profile it determines that
_ZThn24_N7mozilla3net13nsStandardURL16TemplatedMutatorIS1_E4InitEjiRK12nsTSubstringIcEPKcP6nsIURIPP13nsIURIMutator
is likely and so if the vtable is equal to that, it uses special code to run that directly and somehow screws up and passes NULL as this pointer.
If I manually in the debugger on the comparison:
   0x00007fffe55b9032 <+498>:   mov    0x0(%r13),%rax
   0x00007fffe55b9036 <+502>:   lea    0x531463(%rip),%rdx        # 0x7fffe5aea4a0 <_ZThn24_N7mozilla3net13nsStandardURL16TemplatedMutatorIS1_E4InitEjiRK12nsTSubstringIcEPKcP6nsIURIPP13nsIURIMutator>
   0x00007fffe55b903d <+509>:   mov    0x18(%rax),%rax
   0x00007fffe55b9041 <+513>:   cmp    %rdx,%rax
   0x00007fffe55b9044 <+516>:   je     0x7fffe2b330dc <NewStandardURI()-44588388>
change $rdx so that it is not equal to _ZThn24_N7mozilla3net13nsStandardURL16TemplatedMutatorIS1_E4InitEjiRK12nsTSubstringIcEPKcP6nsIURIPP13nsIURIMutator, then it doesn't crash.
 That thunk is
non-virtual thunk to mozilla::net::nsStandardURL::TemplatedMutator<mozilla::net::nsStandardURL>::Init(unsigned int, int, nsTSubstring<char> const&, char const*, nsIURI*, nsIURIMutator**)

Filed upstream, but for now mostly just to track that there is a bug, not what exactly it is.