Bug 2184549 - Firefox 110 crashes shortly after startup with wayland-1.22.0-1.fc39
Summary: Firefox 110 crashes shortly after startup with wayland-1.22.0-1.fc39
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: firefox
Version: rawhide
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Martin Stransky
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 2184743
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-05 01:01 UTC by Adam Williamson
Modified: 2023-04-10 17:11 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-04-10 17:11:33 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Adam Williamson 2023-04-05 01:01:22 UTC
A new wayland build recently landed in Rawhide:

https://bodhi.fedoraproject.org/updates/FEDORA-2023-0e548f86e7

with this build of wayland, Firefox in Rawhide crashes shortly after startup. The same wayland build was sent to F38, but the same crash is *not* happening there.

Rawhide's firefox is actually rather a long way behind F38's: Rawhide has firefox-110.0-3.fc39 , F38 has firefox-111.0.1-1.fc38 . So I suspect this is fixed in firefox 111.0.1 . The original build of 111.0.1 on Rawhide failed due to what looks like some kind of problem in the Rust compile flags:

https://koji.fedoraproject.org/koji/taskinfo?taskID=99049186

+ RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cforce-frame-pointers=yes -Clink-arg=-Wl,-z,relro -Clink-arg=-Wl,-z,now -Clink-arg=%_package_note_flags --cap-lints=warn'
...
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/usr/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin" VSLANG="1033" "cc" "-m64" "/tmp/rustcepiuUM/symbols.o" "/tmp/cargo-installKO9tIo/release/build/getrandom-b797b7dbbd2371e4/build_script_build-b797b7dbbd2371e4.build_script_build.199d6555-cgu.0.rcgu.o" "/tmp/cargo-installKO9tIo/release/build/getrandom-b797b7dbbd2371e4/build_script_build-b797b7dbbd2371e4.3lim40sjeuxgb375.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/cargo-installKO9tIo/release/deps" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-a5e6fbcc934ee18f.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-3b1a85f37d3ea945.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-2a72bd456bf547a2.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-25be9ca95b899fe6.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-583af469e2d2463e.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-b233f187c44a6ba3.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-afc719a420a29c06.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-3e1c0cd488b8bd94.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-c677b7d2ebad01dc.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-8ad1481c13d829cd.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-02c98d239a8cbf47.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-9dda5f81e0e36cd3.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-015c6dfaa32e85de.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-4f0d01b32167fbc6.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-752ab8febd30bed7.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-ab67e71b63c724be.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-5f8e38f6468c2b65.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-0c126a1745191069.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-26c5d6191e5ad5e4.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/cargo-installKO9tIo/release/build/getrandom-b797b7dbbd2371e4/build_script_build-b797b7dbbd2371e4" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs" "-Wl,-z,relro" "-Wl,-z,now" "%_package_note_flags"
  = note: /usr/bin/ld: cannot find %_package_note_flags: No such file or directory
          collect2: error: ld returned 1 exit status

i.e. somehow RUSTFLAGS wound up with the literal string "%_package_note_flags" in it (rather than the value of that macro, which is presumably what it wanted) and that blew up the build.

That's not something the firefox spec itself does, it's coming in from somewhere else. It might have been fixed since, I guess, so we could just retry the firefox build? But we need to do something, or else Firefox in Rawhide will be completely broken with the next compose.

Comment 1 Adam Williamson 2023-04-05 01:40:11 UTC
Sadly, the RUSTFLAGS problem is still present in current Rawhide. Filed: https://bugzilla.redhat.com/show_bug.cgi?id=2184553

at this point we need to find a way to fix it or untag wayland, or we'll be stuck with a broken Firefox...

Comment 2 Adam Williamson 2023-04-05 04:10:19 UTC
Ugh. After working around the first RUSTFLAGS problem, we get this on all arches but s390x:

 0:02.78 Traceback (most recent call last):
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/configure.py", line 343, in <module>
 0:02.78     sys.exit(main(sys.argv))
 0:02.78              ^^^^^^^^^^^^^^
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/configure.py", line 126, in main
 0:02.78     sandbox.run(os.path.join(os.path.dirname(__file__), "moz.configure"))
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/python/mozbuild/mozbuild/configure/__init__.py", line 514, in run
 0:02.78     self._value_for(option)
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/python/mozbuild/mozbuild/configure/__init__.py", line 619, in _value_for
 0:02.78     return self._value_for_option(obj)
 0:02.78            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/python/mozbuild/mozbuild/util.py", line 1061, in method_call
 0:02.78     cache[args] = self.func(instance, *args)
 0:02.78                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/python/mozbuild/mozbuild/configure/__init__.py", line 654, in _value_for_option
 0:02.78     value, option_string = self._helper.handle(option)
 0:02.78                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/python/mozbuild/mozbuild/configure/options.py", line 591, in handle
 0:02.78     ret = option.get_value(arg, origin)
 0:02.78           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 0:02.78   File "/builddir/build/BUILD/firefox-111.0.1/python/mozbuild/mozbuild/configure/options.py", line 448, in get_value
 0:02.78     raise InvalidOptionError(
 0:02.78 mozbuild.configure.options.InvalidOptionError: RUSTFLAGS takes 1 value

this, to me, reads a lot like a bug in Firefox's config parsing stuff. It's all hideously complex, but as best as I can tell, it literally means you can only pass *one* flag. Notably it's not failing on s390x because later in firefox.spec there's this:

%ifarch %{arm} %{ix86} s390x
export RUSTFLAGS="-Cdebuginfo=0"
%endif

(we don't build on 32-bit ARM or i686 any more). That's obviously wrong - it should extend the existing RUSTFLAGS definition, not replace it - but the fact that it's there demonstrates the problem: that definition of RUSTFLAGS is OK because it only has one flag, but the 'normal' (on Rawhide) one with multiple flags makes the build process fail.

This seems to be because RUSTFLAGS is defined with nargs=1:

build/moz.configure/rust.configure:option(env="RUSTFLAGS", nargs=1, help="Rust compiler flags")

when it should probably be * or +. I can try patching it to be +, or we can just undefine RUSTFLAGS in %build to get a build through; this would be no *worse* than the build that's currently tagged in Rawhide...

Comment 3 Adam Williamson 2023-04-05 04:37:39 UTC
ooh. with that change, it fails like this:

0:29.67 error: failed to run `rustc` to learn about target-specific information
 0:29.67 Caused by:
 0:29.67   process didn't exit successfully: `/usr/bin/rustc - --crate-name ___ --print=file-names -Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cforce-frame-pointers=yes -Clink-arg=-Wl -z relro -Clink-arg=-Wl -z now --cap-lints=warn --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
 0:29.67   --- stderr
 0:29.68   error: Unrecognized option: 'z'

note that the commas have gone missing (it's meant to be '-Wl,z,relro'). Which makes me think either those need quoting or escaping somehow, or the Mozilla parser is messing them up?

Comment 4 Martin Stransky 2023-04-05 08:30:59 UTC
Will look at it.

Comment 5 Martin Stransky 2023-04-05 08:47:15 UTC
Updated.

Comment 6 Martin Stransky 2023-04-05 08:47:32 UTC
Err, sorry, wrong bug.

Comment 7 Martin Stransky 2023-04-05 08:57:17 UTC
I testes up-to-date f39/VM and firefox-110.0 which is there by default and I don't see any crashes. May be related to your recent HW, can you provide backtrace of the crash?
https://fedoraproject.org/wiki/Debugging_guidelines_for_Mozilla_products#Using_local_debugging
Thanks.

Comment 8 Adam Williamson 2023-04-05 14:43:33 UTC
Martin: just up-to-date F39 will not show the problem, because we untagged the build so it won't cause this problem. You have to update to the builds from https://koji.fedoraproject.org/koji/buildinfo?buildID=2181261 manually.

Comment 9 Adam Williamson 2023-04-05 14:46:21 UTC
As I expected, your build failed for Rawhide because of the RUSTFLAGS problems:

https://koji.fedoraproject.org/koji/taskinfo?taskID=99553073

 0:04.26 mozbuild.configure.options.InvalidOptionError: RUSTFLAGS takes 1 value

we need to sort the RUSTFLAGS problem out somehow - either fix the upstream issue with parsing it, or just unset RUSTFLAGS at the start of %build - or we will never get a successful build on Rawhide. I can file an upstream issue on the problem at least.

Comment 10 Adam Williamson 2023-04-05 14:50:12 UTC
Oh hey, it has existed for three years(!):

https://bugzilla.mozilla.org/show_bug.cgi?id=1474486

I guess for clarity I'll file yet another bug tracking the build problem, and we can continue the discussion there.

Comment 11 Fabio Valentini 2023-04-05 15:02:38 UTC
> we need to sort the RUSTFLAGS problem out somehow - either fix the upstream issue with parsing it, or just unset RUSTFLAGS at the start of %build

I recommend doing "unset RUSTFLAGS" for now. It restores the status quo ante (i.e. firefox being built without honoring default Rust compiler flags) - which is the same behaviour as on other branches of Fedora.

Comment 12 Adam Williamson 2023-04-05 15:12:18 UTC
Yes, see https://bugzilla.redhat.com/show_bug.cgi?id=2184743 . That's the easy choice, but I kind of want to figure out a reasonable *fix* for this now. I'm working on it. If I can't get something in a couple of hours I'll do the unset thing for now.

Comment 13 Adam Williamson 2023-04-06 03:50:44 UTC
OK, fixed nss, new firefox build running now: https://koji.fedoraproject.org/koji/taskinfo?taskID=99577642

Comment 14 Adam Williamson 2023-04-06 16:05:56 UTC
Aaand the new firefox does fix the problem, so wayland can be retagged, I'll ask releng.

Comment 15 Adam Williamson 2023-04-10 17:11:33 UTC
This is all cleared up now.


Note You need to log in before you can comment on or make changes to this bug.