Bug 1891234

Summary: all tabs crash immediately - firefox unusable
Product: [Fedora] Fedora Reporter: redhatbug
Component: firefoxAssignee: Martin Stransky <stransky>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: awilliam, carlg, david, elxreno, erack, gecko-bugs-nobody, jhorak, jjbn, jld, kai-engert-fedora, law, nixuser, pjasicek, rhughes, rolf.offermanns, rstrode, sandmann, stransky, tqueiros, trpost
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: openqa
Fixed In Version: firefox-82.0-7.fc34 firefox-105.0.2-1.fc38 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 09:48:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
coredump from journalctl
none
firefox fedora binary support info
none
mozilla.org binary support info
none
stack trace
none
strace log
none
strace -ff log
none
Firefox patch none

Description redhatbug 2020-10-24 16:51:47 UTC
Created attachment 1723939 [details]
coredump from journalctl

Description of problem:
On opening of firefox and any tab and immediate crash page appears stating "Gah. Your tab just crashed". This makes firefox unusable.

Version-Release number of selected component (if applicable):
  firefox-82.0-3.fc34.src.rpm

  uname -r: 5.10.0-0.rc0.20201021git071a0578b0ce.49.fc34.x86_64

How reproducible:
Every time firefox or new tab opened.

Steps to Reproduce:
as above

Actual results:
Crashed tab.

Expected results:
Lack of crashed tab.

Additional info:
Starting in safe mode from command line yields the error "/usr/bin/firefox: line 186: [Enforcing: command not found"

Attempted fixes that do not help:
-disabling all plugins/extensions
-safe mode as above
-disabling hardware acceleration
-setting browser.tabs.remote.autostart to false in config
-ensuring system is updated
-moving .mozilla folder in user directory

In journal I found a coredump seemingly relating to libc.so.6 & libmozsandbox.so (is this an SEL-related issue perhaps?) - log attached

Chromium is working fine.

System (just in case relevant): Dell xps15 9500

Comment 1 Martin Stransky 2020-10-24 19:07:38 UTC
Can you try to create a new profile? Run 'firefox -P' on command line.
Thanks.

Comment 2 redhatbug 2020-10-24 19:40:17 UTC
(In reply to Martin Stransky from comment #1)
> Can you try to create a new profile? Run 'firefox -P' on command line.
> Thanks.

Thanks for the response. No change in behavior using a new test profile.

Comment 3 Martin Stransky 2020-10-25 19:40:32 UTC
Okay. Please look at about:crashes page, if there's any crash report please submit it and paste crash ID here.

Please try:

Try X11 backend:
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems?rd=Bug_info_Firefox#Check_Firefox_X11_Gtk.2B_backend_.28Gnome_only.29

Test mozilla binaries:
https://fedoraproject.org/wiki/How_to_debug_Firefox_problems?rd=Bug_info_Firefox#Testing_Mozilla_binaries

Disable Selinux (run setenforce 0 from command line)

Disable Mozilla sandbox, run with MOZ_DISABLE_CONTENT_SANDBOX=1 env variable set on command line.

Comment 4 Martin Stransky 2020-10-25 19:41:22 UTC
Also please check if about:support page works and attach it here.
Thanks.

Comment 5 Carl G. 2020-10-25 23:52:57 UTC
Created attachment 1724092 [details]
firefox fedora binary support info

Comment 6 Carl G. 2020-10-25 23:54:05 UTC
Created attachment 1724093 [details]
mozilla.org binary support info

Comment 7 Carl G. 2020-10-25 23:55:15 UTC
Created attachment 1724094 [details]
stack trace

Comment 8 Carl G. 2020-10-26 00:06:39 UTC
https://crash-stats.mozilla.org/report/index/f75b07b7-48ef-4e78-ac2b-8f1bb0201025#tab-details

I created a new profile to test. Safe mode and refresh doesn't work with the Fedora binary. setenforce 0. Tried w/ both Wayland and X11 session.

Starting Firefox w/ MOZ_DISABLE_CONTENT_SANDBOX=1 works.

Oddly enough, the Mozilla.org binary doesn't crash when opening a website in a new tab and they're both using the same sandbox settings:

Seccomp-BPF (System Call Filtering)	true
Seccomp Thread Synchronization	true
User Namespaces	true
Content Process Sandboxing	true
Media Plugin Sandboxing	true
Content Process Sandbox Level	4
Effective Content Process Sandbox Level	4

I'm attaching the about:support info + stack trace.

Comment 9 Carl G. 2020-10-26 00:11:05 UTC
Just to be clear, MOZ_DISABLE_CONTENT_SANDBOX=1 is only needed on the Fedora Firefox binary.

Comment 10 Martin Stransky 2020-10-26 08:03:15 UTC
Please try to get strace output of FF/Fedora - that may reveal which resource is requested by sandbox. Run as:

strace -o log.txt firefox

and attach the log.txt here.

Thanks.

Comment 11 Rolf Offermanns 2020-10-26 08:21:39 UTC
Created attachment 1724121 [details]
strace log

I see exactly the same behavior as Carl, so here is my strace log.
HTH,
Rolf

Comment 12 Martin Stransky 2020-10-26 08:28:50 UTC
I see, thanks. Looks like the crash happens in a child process so please run strace as:


strace -ff -o log.txt firefox

you'll get more strace files with PID in filename, for each child process one, please attach them as one archive.
Thanks.

Comment 13 Rolf Offermanns 2020-10-26 09:12:10 UTC
Created attachment 1724126 [details]
strace -ff log

Comment 14 Martin Stransky 2020-10-26 11:23:10 UTC
*** Bug 1891449 has been marked as a duplicate of this bug. ***

Comment 15 Martin Stransky 2020-10-26 11:53:27 UTC
Can you please confirm that firefox-82.0-2.fc34.x86_64 build [1] works for you?
Thanks

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=1626345

Comment 16 Jaap 2020-10-26 12:15:00 UTC
 firefox-82.0-2.fc34.x86_64 build [1] works for me on Fedora 34 Rawhide

Comment 17 Rolf Offermanns 2020-10-26 12:24:06 UTC
Yes, this one works for me, too.

BTW: While you are at it. Please fix l.186 in /usr/bin/firefox:

 if [ -x $GETENFORCE_FILE ] && [`getenforce` != "Disabled" ]; then 

There is a space missing before the `getenforce`.

Comment 18 David Hicks 2020-10-26 12:56:11 UTC
firefox-82.0-2.fc34.x86_64.rpm solves this problem for me as well.

Comment 19 redhatbug 2020-10-26 14:28:22 UTC
(In reply to Martin Stransky from comment #3)
> Okay. Please look at about:crashes page, if there's any crash report please
> submit it and paste crash ID here.
> 
> Please try:
> 
> Try X11 backend:
> https://fedoraproject.org/wiki/
> How_to_debug_Firefox_problems?rd=Bug_info_Firefox#Check_Firefox_X11_Gtk.
> 2B_backend_.28Gnome_only.29
> 
> Test mozilla binaries:
> https://fedoraproject.org/wiki/
> How_to_debug_Firefox_problems?rd=Bug_info_Firefox#Testing_Mozilla_binaries
> 
> Disable Selinux (run setenforce 0 from command line)
> 
> Disable Mozilla sandbox, run with MOZ_DISABLE_CONTENT_SANDBOX=1 env variable
> set on command line.

crash report: 75aec27f-db0c-446c-8fdc-3389d0201026

x11 also crashes - crash id: bp-75aec27f-db0c-446c-8fdc-3389d0201026

following sel/sandbox vars did not resolve
setenforce 0
MOZ_DISABLE_CONTENT_SANDBOX=1

haven't yet tried the binaries...

Comment 20 redhatbug 2020-10-26 15:49:11 UTC
Firefox 82.0 binary works for me.

Comment 21 Martin Stransky 2020-10-26 20:05:12 UTC
Okay, I may need to install Rawhide then. I see no significant difference between -2 and -3 builds. May be related to gcc change or some environment setup.

Comment 22 Adam Williamson 2020-10-26 20:11:02 UTC
FWIW, Martin, openQA has been hitting this consistently in Rawhide tests recently (since installer bugs were fixed), and it of course is testing entirely clean installs. I'll see if the next compose with -2 is better.

Comment 23 Jed Davis 2020-10-26 21:29:27 UTC
Firefox bug report: https://bugzilla.mozilla.org/show_bug.cgi?id=1673202

The infinite recursion seen in the attached stacks is because glibc changed `fstat` to call `fstatat` with `AT_EMPTY_PATH` and an empty string; the seccomp-bpf filter can't act on the string contents, so the case is handled in the `SIGSYS` handler, which maps it back to a call to `fstat`.

Older glibc defines `fstat` in the headers as an inline function that calls `__fxstat`, which really is `fstat`, so that would explain why Mozilla's builds work.

This is relatively simple to fix in Firefox, by using the `fstat` (or `fstat64`) syscall directly.

Comment 24 Adam Williamson 2020-10-26 21:35:47 UTC
There's also a couple of active Fedora devel list threads on whether these glibc symbol changes were a good idea...

Paging Jeff Law.

Comment 25 Jed Davis 2020-10-27 00:22:17 UTC
Created attachment 1724427 [details]
Firefox patch

This is the Firefox patch I've sent for review; I'm attaching it here in case it's useful.

I can reproduce the bug locally (building in a Docker container with the Rawhide userland) and have tested the patch that way, and it's passed a subset of tests on the Mozilla try server.

It's not a complete fix, because GeckoMediaPlugin processes (OpenH264, and EME DRM if enabled) currently don't allow `fstatat` at all, so with the new glibc if they call `fstat` it will fail with `ENOSYS` (or, on Nightly, crash with `SIGSYS`).  This part I'm still investigating.

Comment 26 Martin Stransky 2020-10-27 09:35:41 UTC
Thanks a lot Jed!

Comment 27 Martin Stransky 2020-10-27 09:51:04 UTC
Added to firefox-82.0-7.fc34

Comment 28 Adam Williamson 2020-10-28 16:05:20 UTC
For the record, that build failed, and today's Rawhide still has 82.0-3.fc34, and still has the bug.

82.0.1-1 succeeded, though - https://koji.fedoraproject.org/koji/buildinfo?buildID=1635450 - so should be in the next compose and we'll hope this is fixed.

Comment 29 Ian Laurie 2020-10-28 21:09:15 UTC
82.0.1-1 downloaded from the koji link above has fixed it for me.

Comment 30 Fedora Update System 2022-10-05 12:47:02 UTC
FEDORA-2022-f0988ea008 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f0988ea008

Comment 31 Fedora Update System 2022-10-05 12:56:51 UTC
FEDORA-2022-f0988ea008 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.