Bug 2226990

Summary: Firefox build failure: error: could not compile `gkrust` (lib), (signal: 9, SIGKILL: kill)
Product: [Fedora] Fedora Reporter: Martin Stransky <stransky>
Component: rustAssignee: Rust SIG <rust-sig>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 38CC: amulhern, decathorpe, igor.raits, jistone, rust-sig, TicoTimo
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Stransky 2023-07-27 10:06:59 UTC
Firefox 116.0 fails to build while 115.0 proceed fine.
It affects Fedora 38 only, Fedora 37 built OK with the same config.

Task:
https://koji.fedoraproject.org/koji/taskinfo?taskID=103947497

Build log:
https://kojipkgs.fedoraproject.org//work/tasks/7497/103947497/build.log

25:14.95 error: could not compile `gkrust` (lib)
25:14.95 Caused by:
25:14.95   process didn't exit successfully: `CARGO=/usr/bin/cargo CARGO_CRATE_NAME=gkrust CARGO_MANIFEST_DIR=/builddir/build/BUILD/firefox-116.0/toolkit/library/rust CARGO_PKG_AUTHORS='The Mozilla Project Developers' CARGO_PKG_DESCRIPTION='Rust code for libxul' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=gkrust CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/builddir/build/BUILD/firefox-116.0/objdir/instrumented/release/deps:/usr/lib' /usr/bin/rustc --crate-name gkrust toolkit/library/rust/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type staticlib --emit=dep-info,link -C opt-level=2 -C panic=abort -C embed-bitcode=no -Clto --cfg 'feature="mozilla-central-workspace-hack"' -C metadata=0db929ed760f375e -C extra-filename=-0db929ed760f375e --out-dir /builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C linker=/builddir/build/BUILD/firefox-116.0/build/cargo-linker -L dependency=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps -L dependency=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/release/deps --extern gkrust_shared=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libgkrust_shared-d4d7db70aea87b4d.rlib --extern lmdb_sys=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/liblmdb_sys-252f9361a5999db0.rlib --extern mozglue_static=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozglue_static-88ece5d5c6136c62.rlib --extern mozilla_central_workspace_hack=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libmozilla_central_workspace_hack-81cb987ea3ad6fe0.rlib --extern swgl=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/deps/libswgl-f6abc6f7633bf65c.rlib -C debuginfo=2 --cap-lints warn -Cembed-bitcode=yes -C codegen-units=1 -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/audioipc2-b10544885229944e/out -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/lmdb-rkv-sys-fa9e335369c29fa0/out -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/mozglue-static-e90b6631bd318db4/out -L native=/usr/lib64 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/dist/bin -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/nss/nss_nss3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/security/nss/lib/ssl/ssl_ssl3 -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/config/external/nspr/pr -L native=/builddir/build/BUILD/firefox-116.0/objdir/instrumented/x86_64-unknown-linux-gnu/release/build/swgl-ff81364e355f9cbb/out` (signal: 9, SIGKILL: kill)
25:16.15 gmake[4]: *** [/builddir/build/BUILD/firefox-116.0/config/makefiles/rust.mk:434: force-cargo-library-build] Error 101
25:16.15 gmake[4]: Leaving directory '/builddir/build/BUILD/firefox-116.0/objdir/instrumented/toolkit/library/rust'
25:16.15 gmake[3]: *** [/builddir/build/BUILD/firefox-116.0/config/recurse.mk:72: toolkit/library/rust/target] Error 2
25:16.15 gmake[3]: *** Waiting for unfinished jobs....


Reproducible: Always

Comment 1 Fabio Valentini 2023-07-27 10:31:34 UTC
This looks like rustc is getting reaped by the OOM killer ...

Can you check whether your build on F37 used Rust 1.71 or still 1.70? If it's the latter, then this looks like a new problem with Rust 1.71.

Comment 2 Martin Stransky 2023-07-27 11:11:52 UTC
Looks like so. Latest successful Firefox 115.0.2 build for Fedora 38 was done with rustc version 1.70.0 on Thu, 20 Jul:
https://kojipkgs.fedoraproject.org//packages/firefox/115.0.2/3.fc38/data/logs/x86_64/build.log

But recent builds with rustc version 1.71.0 fails (build config is almost the same).

Comment 3 Josh Stone 2023-07-27 19:29:41 UTC
I see you now have a successful build on f38 with 1.71.0 -- what changed?
https://koji.fedoraproject.org/koji/buildinfo?buildID=2264243

Comment 4 Martin Stransky 2023-07-28 06:53:02 UTC
(In reply to Josh Stone from comment #3)
> I see you now have a successful build on f38 with 1.71.0 -- what changed?
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2264243

Nothing significant - just changelog update and rawhide python build fix.
If the failure is caused by OOM maybe different build box with more ram was chosen by koji?

Comment 5 Josh Stone 2023-07-28 19:34:03 UTC
Hmm, yes, the failed build was on buildvm-x86-07 with 16 GiB, and it succeeded on buildhw-x86-02 with 128 GiB.

Maybe you can reduce parallelism when there's less memory available? Or invoke the gkrust build separately, before the full build?

(Assuming that killed process didn't consume all memory on its own -- LTO might do that! But it looks like other C++ processes were active too.)

Comment 6 Fabio Valentini 2023-07-28 19:47:34 UTC
Since you're already using the %constrain_build macro for firefox:
%constrain_build -m 2048
It might be worth trying to set this to a higher value?

Comment 7 Martin Stransky 2023-08-02 11:04:26 UTC
%constrain_build -m 4000 seems to be working (somehow).