Firefox 116.0 fails to build while 115.0 proceed fine. It affects Fedora 38 only, Fedora 37 built OK with the same config. Task: https://koji.fedoraproject.org/koji/taskinfo?taskID=103947497 Build log: https://kojipkgs.fedoraproject.org//work/tasks/7497/103947497/build.log 25:14.95 error: could not compile `gkrust` (lib) 25:14.95 Caused by: 25:14.95 process didn't exit successfully: `CARGO=/usr/bin/cargo CARGO_CRATE_NAME=gkrust CARGO_MANIFEST_DIR=/builddir/build/BUILD/firefox-116.0/toolkit/library/rust CARGO_PKG_AUTHORS='The Mozilla Project Developers' CARGO_PKG_DESCRIPTION='Rust code for libxul' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=gkrust CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/builddir/build/BUILD/firefox-116.0/objdir/instrumented/release/deps:/usr/lib' /usr/bin/rustc --crate-name gkrust toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C embed-bitcode=no -Clto --cfg 'feature="mozilla-central-workspace-hack"' -C metadata=0db929ed760f375e -C extra-filename=-0db929ed760f375e --out-dir /builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=/builddir/build/BUILD/firefox-116.0/build/cargo-linker -L dependency=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps -L dependency=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/release/deps --extern gkrust_shared=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libgkrust_shared-d4d7db70aea87b4d.rlib --extern lmdb_sys=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/liblmdb_sys-252f9361a5999db0.rlib --extern mozglue_static=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozglue_static-88ece5d5c6136c62.rlib --extern mozilla_central_workspace_hack=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozilla_central_workspace_hack-81cb987ea3ad6fe0.rlib --extern swgl=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libswgl-f6abc6f7633bf65c.rlib -C debuginfo=2 --cap-lints warn -Cembed-bitcode=yes -C codegen-units=1 -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/audioipc2-b10544885229944e/out -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/lmdb-rkv-sys-fa9e335369c29fa0/out -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/mozglue-static-e90b6631bd318db4/out -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/swgl-ff81364e355f9cbb/out` (signal: 9, SIGKILL: kill) 25:16.15 gmake[4]: *** [/builddir/build/BUILD/firefox-116.0/config/makefiles/rust.mk:434: force-cargo-library-build] Error 101 25:16.15 gmake[4]: Leaving directory '/builddir/build/BUILD/firefox-116.0/objdir/instrumented/toolkit/library/rust' 25:16.15 gmake[3]: *** [/builddir/build/BUILD/firefox-116.0/config/recurse.mk:72: toolkit/library/rust/target] Error 2 25:16.15 gmake[3]: *** Waiting for unfinished jobs.... Reproducible: Always
This looks like rustc is getting reaped by the OOM killer ... Can you check whether your build on F37 used Rust 1.71 or still 1.70? If it's the latter, then this looks like a new problem with Rust 1.71.
Looks like so. Latest successful Firefox 115.0.2 build for Fedora 38 was done with rustc version 1.70.0 on Thu, 20 Jul: https://kojipkgs.fedoraproject.org//packages/firefox/115.0.2/3.fc38/data/logs/x86_64/build.log But recent builds with rustc version 1.71.0 fails (build config is almost the same).
I see you now have a successful build on f38 with 1.71.0 -- what changed? https://koji.fedoraproject.org/koji/buildinfo?buildID=2264243
(In reply to Josh Stone from comment #3) > I see you now have a successful build on f38 with 1.71.0 -- what changed? > https://koji.fedoraproject.org/koji/buildinfo?buildID=2264243 Nothing significant - just changelog update and rawhide python build fix. If the failure is caused by OOM maybe different build box with more ram was chosen by koji?
Hmm, yes, the failed build was on buildvm-x86-07 with 16 GiB, and it succeeded on buildhw-x86-02 with 128 GiB. Maybe you can reduce parallelism when there's less memory available? Or invoke the gkrust build separately, before the full build? (Assuming that killed process didn't consume all memory on its own -- LTO might do that! But it looks like other C++ processes were active too.)
Since you're already using the %constrain_build macro for firefox: %constrain_build -m 2048 It might be worth trying to set this to a higher value?
%constrain_build -m 4000 seems to be working (somehow).