Created attachment 1865895 [details] output of 'journalctl -a' Description of problem: When the user attempts to open Firefox, either through the terminal or gnome, Firefox never opens and the user is shown an endlessly-spinning wheel. Log is littered with the following messages: fedora gnome-shell[1689]: Received error from D-Bus search provider firefox.desktop: Gio.DBusError: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name is not activatable Version-Release number of selected component (if applicable): Fedora-Workstation-36_Beta-1.1.aarch64.raw.xz firefox-98.0-2.fc36.aarch64 How reproducible: Always Steps to Reproduce: 1. Install Fedora-Workstation-36_Beta-1.1.aarch64.raw.xz 2. Attempt to open Firefox, either through terminal or GUI 3. See spinning wheel and Firefox never opens Additional info: See attached log.
Proposed as a Blocker for 36-final by Fedora user coremodule using the blocker tracking app because: Proposing as a possible violation of the following F36 Final criterion: "For all release-blocking desktop / arch combinations, the following applications must start successfully and withstand a basic functionality test... web browser." [0] [0] https://fedoraproject.org/wiki/Fedora_36_Final_Release_Criteria#Default_application_functionality
Created attachment 1865909 [details] terminal output when attempting to run firefox from CLI
Martin, can you take a look at this urgently? We're accepting it as a Beta blocker, and go/no-go is Thursday - I was all set to run a candidate compose in a couple of hours till we found out about this. Thanks!
sallyahaj reported on the update - https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d#comment-2443196 - that on Xfce a window opens but with nothing inside it, which makes me think those "In pixman_region32_init_rect: Invalid rectangle passed" errors are the issue here. The cgroupify stuff is probably https://bugzilla.redhat.com/show_bug.cgi?id=2057184 which we've been told is not critical, I think.
Discussed during the 2022-03-14 blocker review meeting: [0] The decision to classify this bug as an "AcceptedBlocker (Beta)" was made as it violates the following Basic criterion: "It must be possible to run the default web browser and a terminal application from all release-blocking desktop environments". Note this bug is introduced by Firefox 98 which is not yet stable, but including 98 is itself a blocker [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-03-14/f36-blocker-review.2022-03-14-16.01.txt
Is there any aarch64 workstation or can I run that emulated on x86_64 somehow? Also can you try to kill firefox when you see the wheel and paste backtrace? (see https://fedoraproject.org/wiki/Debugging_guidelines_for_Mozilla_products#Application_freeze)
Created attachment 1865933 [details] screencast of what happens when Firefox is clicked on a freshly-booted system Here is a screencast of the affected system. The spinning wheel persists even after the Firefox process is killed.
Created attachment 1865934 [details] backtrace of firefox Backtrace from [baggypants@eizouken ~]$ rpm -qi firefox-debuginfo Name : firefox-debuginfo Version : 98.0 Release : 2.fc35 Architecture: aarch64 Install Date: Mon 14 Mar 2022 23:03:49 GMT Group : Development/Debug Size : 2739573032 License : MPLv1.1 or GPLv2+ or LGPLv2+ Signature : RSA/SHA256, Tue 08 Mar 2022 18:01:07 GMT, Key ID db4639719867c58f Source RPM : firefox-98.0-2.fc35.src.rpm Build Date : Sat 05 Mar 2022 08:13:29 GMT Build Host : buildvm-a64-30.iad2.fedoraproject.org
Martin: Jetson Nano or Raspberry Pi 4 are the most common platforms for testing Workstation on aarch64, if you have one of those. You can run it in emulation but it would be extremely slow as aarch64-on-x86_64 is pure software emulation; you'd just have to install the relevant qemu and libvirt packages, then you will be able to create an aarch64 VM in virt-manager, but it'll run 10x or 20x slower than a real system would. Let us know if the backtrace from L.L. Robinson is good, if not, we'll try and get another. Thanks!
(In reply to Adam Williamson from comment #9) > Martin: Jetson Nano or Raspberry Pi 4 are the most common platforms for > testing Workstation on aarch64, if you have one of those. You can run it in > emulation but it would be extremely slow as aarch64-on-x86_64 is pure > software emulation; you'd just have to install the relevant qemu and libvirt > packages, then you will be able to create an aarch64 VM in virt-manager, but > it'll run 10x or 20x slower than a real system would. Unfortunately I don't have any such hardware. The backtrace is clear: #4 0x0000fffff7fc92a0 in _dl_open (file=0xffffffffcae8 "/usr/lib64/firefox/libxul.so", mode=-2147483391, caller_dlopen=0xaaaaaaaaceec <XPCOMGlueLoad(char const*, mozilla::LibLoadingStrategy)+524>, nsid=-2, argc=1, argv=0xffffffffed78, env=0xaaaaaab58230) at dl-open.c:896 args = {file = 0xffffffffcae8 "/usr/lib64/firefox/libxul.so", mode = -2147483391, caller_dlopen = 0xaaaaaaaaceec <XPCOMGlueLoad(char const*, mozilla::LibLoadingStrategy)+524>, map = 0xaaaaaab83980, nsid = 0, original_global_scope_pending_adds = 0, libc_already_loaded = true, worker_continue = false, argc = 1, argv = 0xffffffffed78, env = 0xaaaaaab58230} exception = {objname = 0xc97 <error: Cannot access memory at address 0xc97>, errstring = 0xc9900000000 <error: Cannot access memory at address 0xc9900000000>, message_buffer = 0x0} errcode = <optimized out> __PRETTY_FUNCTION__ = "_dl_open" <error: Cannot access memory at address 0xc97>, errstring = 0xc9900000000 <error: Cannot access memory at address 0xc9900000000>, message_buffer = 0x0} Is is possible that libxul.so is corrupted? Moving to glibc to get insight from toolchain people (but that may not be a bug in glibc/dlopen).
(In reply to Adam Williamson from comment #9) > Martin: Jetson Nano or Raspberry Pi 4 are the most common platforms for > testing Workstation on aarch64, if you have one of those. I have a Raspberry Pi 4, but the Fedora wiki says it's not fully working. So I wonder why this is considered a release blocker. https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi#Raspberry_Pi_4 If it's not expected to work, I'm not sure how I'm supposed to use it to debug aarch64 issues. How would I know that the problems I see a genuine regressions, or just not expected to work yet? (In reply to Martin Stransky from comment #10) > The backtrace is clear: > > #4 0x0000fffff7fc92a0 in _dl_open (file=0xffffffffcae8 > "/usr/lib64/firefox/libxul.so", mode=-2147483391, > caller_dlopen=0xaaaaaaaaceec <XPCOMGlueLoad(char const*, > mozilla::LibLoadingStrategy)+524>, nsid=-2, argc=1, argv=0xffffffffed78, > env=0xaaaaaab58230) at dl-open.c:896 > args = {file = 0xffffffffcae8 "/usr/lib64/firefox/libxul.so", mode = > -2147483391, caller_dlopen = 0xaaaaaaaaceec <XPCOMGlueLoad(char const*, > mozilla::LibLoadingStrategy)+524>, map = 0xaaaaaab83980, nsid = 0, > original_global_scope_pending_adds = 0, libc_already_loaded = true, > worker_continue = false, argc = 1, argv = 0xffffffffed78, env = > 0xaaaaaab58230} > exception = {objname = 0xc97 <error: Cannot access memory at address > 0xc97>, errstring = 0xc9900000000 <error: Cannot access memory at address > 0xc9900000000>, message_buffer = 0x0} > errcode = <optimized out> > __PRETTY_FUNCTION__ = "_dl_open" > <error: Cannot access memory at address 0xc97>, errstring = 0xc9900000000 > <error: Cannot access memory at address 0xc9900000000>, message_buffer = 0x0} > > Is is possible that libxul.so is corrupted? > > Moving to glibc to get insight from toolchain people (but that may not be a > bug in glibc/dlopen). Unlikely. This bug was filed for Fedora 36, but the line number reference does not match Fedora 36's glibc (dl-open.c:896 is in the middle of a multi-line comment). The file hasn't recently changed. And is Firefox really stuck in glibc, or is it perhaps calling dlopen in an endless loop?
Sorry if this wasn't clear. This is also affecting Firefox 98 on Fedora 35 (98.0-2.fc35) which the backtrace was taken on, using a Pinebook Pro RK3399.
(In reply to L.L.Robinson from comment #12) > Sorry if this wasn't clear. This is also affecting Firefox 98 on Fedora 35 > (98.0-2.fc35) which the backtrace was taken on, using a Pinebook Pro RK3399. Oh. Is it really hanging on this line (dl-open.c:767 is at the top of the backtrace), or is the function exiting? You can continue execution until the function returns using the “finish” command. Run it several times to see if control gets returned to Firefox. Thanks.
Sorry, I have no idea. Firefox doesn't crash, so I presume no function is exiting. I'm getting Firefox to drop to the gdb command prompt by sending 'kill -s 11 $(pidof firefox)', then running a backtrace from there. Also, this upstream bug may be relevent https://bugzilla.mozilla.org/show_bug.cgi?id=1757571
You can attach to the running Firefox using: gdb -p $(pidof firefox) There isn't much detail in the upstream bug, so it is hard to tell if it's the same issue. One way to check would be to revert the relevant commit and see if the problem is gone.
Attaching gdb to a running firefox throws up a continuous amount of: BFD: /usr/lib/debug/usr/lib64/firefox/libxul.so-98.0-2.fc35.aarch64.debug: attempt to load strings from a non-string section (number 41) I'll follow through https://fedoraproject.org/wiki/Debugging_guidelines_for_Mozilla_products#Application_freeze and get a backtrace.
Created attachment 1865994 [details] Another backtrace on firefox-debuginfo-98.0-2.fc35.aarch64 firefox-debuginfo-98.0-2.fc35.aarch64 collect on Pinebook Pro attached to firefox using `gdb -p $(pidof firefox) followed https://fedoraproject.org/wiki/Debugging_guidelines_for_Mozilla_products#Application_freeze
(In reply to L.L.Robinson from comment #17) > Created attachment 1865994 [details] > Another backtrace on firefox-debuginfo-98.0-2.fc35.aarch64 > > firefox-debuginfo-98.0-2.fc35.aarch64 > > collect on Pinebook Pro > > attached to firefox using `gdb -p $(pidof firefox) > > followed > https://fedoraproject.org/wiki/ > Debugging_guidelines_for_Mozilla_products#Application_freeze This latest backtrace doesn't show dlopen anymore. Should we reassign back to firefox?
(In reply to L.L.Robinson from comment #17) > Created attachment 1865994 [details] > Another backtrace on firefox-debuginfo-98.0-2.fc35.aarch64 > > firefox-debuginfo-98.0-2.fc35.aarch64 > > collect on Pinebook Pro > > attached to firefox using `gdb -p $(pidof firefox) > > followed > https://fedoraproject.org/wiki/ > Debugging_guidelines_for_Mozilla_products#Application_freeze This backtrace seems to come from child process (web content process). You need to get bt from main Firefox process (the topmost one).
L.L.Robinson, can you please run Firefox in gdb directly and attach the bt here? see: https://fedoraproject.org/wiki/Debugging_guidelines_for_Mozilla_products#Running_application_in_debugger
That's how I created the initial backtrace Martin. I'll run another one for you though.
(In reply to Florian Weimer from comment #11) > I have a Raspberry Pi 4, but the Fedora wiki says it's not fully working. So > I wonder why this is considered a release blocker. > This bug seems to be affecting F36 aarch64 across all aarch64 hardware, not just on the RPi4, which is why it's a release blocker. I personally know it appears on both Jetson Nano and RPi4 hardware. > https://fedoraproject.org/wiki/Architectures/ARM/Raspberry_Pi#Raspberry_Pi_4 > > If it's not expected to work, I'm not sure how I'm supposed to use it to > debug aarch64 issues. How would I know that the problems I see a genuine > regressions, or just not expected to work yet? In general, Fedora on RPi4 works fine and as expected. It is considered "unsupported" in that we wouldn't necessarily block the release on a specific-to-RPi4 bug, should one arise. We take that on a case-by-case basis.
stransky: can we try a build with the rust crate downgrade mentioned in the upstream issue? How complex would that be to implement?
(In reply to Adam Williamson from comment #23) > stransky: can we try a build with the rust crate downgrade mentioned in the > upstream issue? How complex would that be to implement? We need to revert https://bugzilla.mozilla.org/show_bug.cgi?id=1750646 but dunno if it's buildable. I'll prefer to identify exact cause so we need another backtrace - https://bugzilla.redhat.com/show_bug.cgi?id=2063961#c21
Created attachment 1866107 [details] 'firefox -g -d gdb' Martin, try this.
Same issue here in rpi4 with XFCE on 36 Beta release.
(In reply to Geoffrey Marr from comment #25) > Created attachment 1866107 [details] > 'firefox -g -d gdb' > > Martin, try this. Looks like the dlopen issue, Thanks.
Fyi, while Firefox 97 was working, after this weeks update to Firefox 98 i see this same issue too and Firefox no longer starts: Mar 16 08:16:39 fedora gnome-shell[1475]: Received error from D-Bus search provider firefox.desktop: Gio.DBusError: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name is not activatable Firefox 98.0-2.fc35 Fedora Linux 35 (Workstation Edition) Linux 5.11.12-300.fc34.aarch64
I fired new builds with downgraded crates (https://bugzilla.mozilla.org/show_bug.cgi?id=1750646) - firefox-98.0-3.fc35
We can't build FF on F36 due to missing updated GCC there - https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d But F37 builds seems to be OK so you may test them on aarch64.
ah, yeah, I didn't push the update stable because of this problem! it just needs a buildroot override, Frantisek has created one, you should be able to cancel the current task and run another build now that will work.
OK, I ran a build now.
FEDORA-2022-154de14def has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-154de14def
FEDORA-2022-154de14def has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
F37 update shouldn't have been marked as fixing this.
(In reply to Fedora Update System from comment #33) > FEDORA-2022-154de14def has been submitted as an update to Fedora 37. > https://bodhi.fedoraproject.org/updates/FEDORA-2022-154de14def Tested on RPi4 with Fedora-Workstation-36_Beta-1.1.aarch64.raw.xz. Works as it should, tried general browsing, YouTube, logging in to FAS. All work great. LGTM.
Yep, firefox-98.0-3.fc37 works nicely both on rpi4 and Pinebook.
FEDORA-2022-42ea499a7d has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d
firefox-98.0-3.fc36 works with me on rpi4. Thank you.
FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-42ea499a7d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-42ea499a7d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-42ea499a7d has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.
Are you aware this is also still an issue in Fedora 35? Will this update be pushed there? firefox.aarch64 98.0-2.fc35 Fedora Linux 35 (Workstation Edition) Linux 5.11.12-300.fc34.aarch64
It's already built for F34 and F35, not sure if there's a reason Martin didn't submit an update yet, but you can get it from Koji if you want it right away.
FEDORA-2022-026d7123e8 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-026d7123e8
FEDORA-2022-dcd7de0648 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-dcd7de0648
FEDORA-2022-dcd7de0648 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-dcd7de0648` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-dcd7de0648 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-026d7123e8 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-026d7123e8` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-026d7123e8 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-dcd7de0648 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-76576598bb has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-76576598bb
FEDORA-2022-76576598bb has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-6a9fba03f9 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-6a9fba03f9` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-6a9fba03f9 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-6a9fba03f9 has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-f0988ea008 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f0988ea008
FEDORA-2022-f0988ea008 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.