I can reproduce it locally on x86_64 too when PGO+LTO is enabled.

What's the status of this bug?

Workstation/aarch64/images/Fedora-Workstation-aarch64-_RELEASE_MILESTONE_-sda.raw.xz is a release blocking image. If bug 1881495 accepted as freeze exception or blocker; then this bug becomes a blocker.

Could we re-enable aarch64 and attempt another build with all the changes? Minimally this will give us logs for F33

Let's see if the new builds fixes that.

This is a pre-requirement for bug 1881495, proposing for a freeze exception as well.

I can't reproduce that on x86_64 any more. If that's aarch64 specific I have no idea how to fix that.

Let's see how https://koji.fedoraproject.org/koji/taskinfo?taskID=52052089 goes.

For some reasons mozgtk/gtk2/libmozgtk.so and mozgtk/gtk3/libmozgtk.so directories are ignored on aarch64.

Correct build log (x86_64): https://kojipkgs.fedoraproject.org//work/tasks/2240/52052240/build.log
Broken one (aarch64): https://kojipkgs.fedoraproject.org//work/tasks/2241/52052241/build.log

The libmozgtk.so build task below is completelly missing on aarch64:

70:29.40 gmake[4]: Entering directory '/builddir/build/BUILD/firefox-81.0/objdir/widget/gtk/mozgtk/gtk2'
70:29.40 widget/gtk/mozgtk/gtk2/libmozgtk.so
70:29.40 rm -f libmozgtk.so
70:29.40 /usr/bin/gcc -std=gnu99 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fpermissive -fPIC -Wl,-z,relro -Wl,-z,now -DNSS_PKCS11_3_0_STRICT -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe -g -freorder-blocks -O2 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wduplicated-cond -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wno-error=deprecated-copy -Wformat -Wformat-overflow=2 -Werror=implicit-function-declaration -Wno-psabi  -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libmozgtk.so -o libmozgtk.so mozgtk.o   -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -fstack-protector-strong -Wl,-rpath-link,/builddir/build/BUILD/firefox-81.0/objdir/dist/bin -Wl,-rpath-link,/usr/lib      -ldl  -Wl,--no-as-needed -lgtk-x11-2.0 -lgdk-x11-2.0 -Wl,--as-needed
70:29.42 gmake[4]: Leaving directory '/builddir/build/BUILD/firefox-81.0/objdir/toolkit/crashreporter/minidump-analyzer'
70:29.43 gmake[4]: Entering directory '/builddir/build/BUILD/firefox-81.0/objdir/widget/gtk/mozgtk/gtk3'
70:29.43 widget/gtk/mozgtk/gtk3/libmozgtk.so
70:29.43 rm -f libmozgtk.so
70:29.43 /usr/bin/gcc -std=gnu99 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fpermissive -fPIC -Wl,-z,relro -Wl,-z,now -DNSS_PKCS11_3_0_STRICT -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe -g -freorder-blocks -O2 -fomit-frame-pointer -funwind-tables -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wduplicated-cond -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wno-error=deprecated-copy -Wformat -Wformat-overflow=2 -Werror=implicit-function-declaration -Wno-psabi  -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libmozgtk.so -o libmozgtk.so mozgtk.o   -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -fstack-protector-strong -Wl,-rpath-link,/builddir/build/BUILD/firefox-81.0/objdir/dist/bin -Wl,-rpath-link,/usr/lib      -ldl  -Wl,--no-as-needed -lgtk-3 -lgdk-3 -Wl,--as-needed
70:29.46 /builddir/build/BUILD/firefox-81.0/objdir/_virtualenvs/init_py3/bin/python -m mozbuild.action.check_binary --target libmozgtk.so
70:29.46 /builddir/build/BUILD/firefox-81.0/objdir/_virtualenvs/init_py3/bin/python -m mozbuild.action.check_binary --target libmozgtk.so
70:29.60 chmod +x libmozgtk.so
70:29.60 ../../../../config/nsinstall -R -m 644 'libmozgtk.so' '../../../../dist/bin'
70:29.60 gmake[4]: Leaving directory '/builddir/build/BUILD/firefox-81.0/objdir/widget/gtk/mozgtk/gtk3'
70:29.61 /usr/bin/g++ -std=gnu++17 -o TelemetryIOInterposeObserver.o -c  -I/builddir/build/BUILD/firefox-81.0/objdir/dist/stl_wrappers -I/builddir/build/BUILD/firefox-81.0/objdir/dist/system_wrappers -include /builddir/build/BUILD/firefox-81.0/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -DNDEBUG=1 -DTRIMMED=1 -DOS_POSIX=1 -DOS_LINUX=1 '-DMOZ_APP_VERSION="81.0"' -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builddir/build/BUILD/firefox-81.0/toolkit/components/telemetry -I/builddir/build/BUILD/firefox-81.0/objdir/toolkit/components/telemetry -I/builddir/build/BUILD/firefox-81.0/objdir/ipc/ipdl/_ipdlheaders -I/builddir/build/BUILD/firefox-81.0/ipc/chromium/src -I/builddir/build/BUILD/firefox-81.0/ipc/glue -I/builddir/build/BUILD/firefox-81.0/xpcom/build -I/builddir/build/BUILD/firefox-81.0/xpcom/threads -I/builddir/build/BUILD/firefox-81.0/objdir/dist/include -I/usr/include/nspr4 -I/usr/include/nss3 -I/usr/include/nspr4 -I/builddir/build/BUILD/firefox-81.0/objdir/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builddir/build/BUILD/firefox-81.0/objdir/mozilla-config.h -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wno-invalid-offsetof -Wc++2a-compat -Wduplicated-cond -Wimplicit-fallthrough -Wunused-function -Wunused-variable -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=coverage-mismatch -Wno-error=free-nonheap-object -Wno-multistatement-macros -Wno-error=class-memaccess -Wno-error=deprecated-copy -Wformat -Wformat-overflow=2 -Werror=implicit-function-declaration -Wno-psabi -fno-sized-deallocation -fno-aligned-new -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fpermissive -fPIC -Wl,-z,relro -Wl,-z,now -DNSS_PKCS11_3_0_STRICT -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -freorder-blocks -O2 -fomit-frame-pointer -funwind-tables -Wno-error=shadow  -MD -MP -MF .deps/TelemetryIOInterposeObserver.o.pp   /builddir/build/BUILD/firefox-81.0/toolkit/components/telemetry/other/TelemetryIOInterposeObserver.cpp
70:29.61 cc1plus: warning: ‘-Werror=’ argument ‘-Werror=implicit-function-declaration’ is not valid for C++
70:29.61 chmod +x libmozgtk.so
70:29.61 ../../../../config/nsinstall -R -m 644 'libmozgtk.so' '../../../../dist/bin/gtk2'
70:29.61 gmake[4]: Leaving directory '/builddir/build/BUILD/firefox-81.0/objdir/widget/gtk/mozgtk/gtk2'

Hm, the builds happens at widget/gtk/mozgtk directory. I see only stub library is buidt, gtk2/gtk3 subdirs are ignored. I wonder if it's caused by the different moz.build files at gtk3/gtk2 where the library setup is configured regards the LD flags. That may be the factor here.

I think I stumbled upon a fix for this: enabling parallel builds :)

I don't know what's going on, but mozgtk seems to get correctly built at least. I guess there must be some kind of undeclared dependency/race in the build system and when building in parallel it manages to build mozgtk in time for the next step that needs it.

https://src.fedoraproject.org/rpms/firefox/c/bd0f8654262746237583ab12c0f0d53fb496a81e?branch=f33

Let's see how https://koji.fedoraproject.org/koji/taskinfo?taskID=52099114 goes.

FEDORA-2020-99834af551 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-99834af551

FEDORA-2020-99834af551 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-99834af551`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-99834af551

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-99834af551 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.