Description of problem: I have updated to firefox-69.0.1-1.fc30 as part of https://bodhi.fedoraproject.org/updates/FEDORA-2019-2b6fab9eb9 and fully rebooted. Now all operations in Firefox feel slower and sluggish. Starting the browser feels slower, using the UI (opening new tabs, using the address bar) feels slower, and using modern heavy pages (like gmail, facebook, patreon) feels much slower. On facebook, I feel like the rendering is several times slower. I tried to downgrade to firefox-69.0-2.fc30 and it feels better again. And back and forth. I don't know how to provide measurements, perhaps there is some benchmark I can run to provide comparisons? Also, in system journal, I see *thousands* of messages like this: Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir/build/BUILD/firefox-69.0.1/objdir/mozglue/misc/Mutex_posix.gcda:Skip Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir:Cannot create directory Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir/build/BUILD/firefox-69.0.1/objdir/mozglue/misc/ConditionVariable_posix.gcda:Skip Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir:Cannot create directory Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir/build/BUILD/firefox-69.0.1/objdir/mozglue/misc/AutoProfilerLabel.gcda:Skip Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir:Cannot create directory Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir/build/BUILD/firefox-69.0.1/objdir/mozglue/baseprofiler/Unified_cpp_mozglue_baseprofiler1.gcda:Skip Sep 20 11:00:40 phoenix firefox.desktop[2095]: profiling:/builddir:Cannot create directory Perhaps some profiling/debugging option has been left on by accident? Note that I'm using Fedora 30 with X11 session. My hardware: Thinkpad T480s 00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics 620 [8086:5917] (rev 07) Version-Release number of selected component (if applicable): firefox-69.0.1-1.fc30.x86_64 - feels slow firefox-69.0-2.fc30.x86_64 - feels better How reproducible: always Steps to Reproduce: 1. compare using sites like facebook on the old and the new version. Also compare just using the UI. Additional info: It might be just a feeling on my side.
I see, Thanks for the report. There's a similar one - Bug 1753372. Can you please your about:support page? Also can you try to: - Use gecko-profiler to profile the performance: https://profiler.firefox.com/ - Check "about:performance" page to see if there's any page/addon consumes the CPU - Try Wayland version (Run Gnome on Wayland, install firefox-wayland and run it) Thanks.
Also can you try to create and test a new profile?
*** Bug 1753372 has been marked as a duplicate of this bug. ***
profiling:/builddir/build/BUILD/firefox-69.0.1/objdir/mozglue/misc/Mutex_posix.gcda:Skip is a clear indication that the second run of PGO did not run and you have a binary with the profiling instructions!
I didn't know that, Thanks. It should be fixed by https://bodhi.fedoraproject.org/updates/FEDORA-2019-f7d21d6ef6
firefox-69.0.1-3.fc30.x86_64 is way better PGO is simple: the first build "spits" collecting code to the binary and by running that binary with code-coverage the gcda files have records with codepath are hot to give the compiler for the second compile optimization informations which are not available otherwise, so a PGO build with only the frist run is slow by definition because it's not optimized at all and full of collecting code doing that for PHP since 2016 by running the test-suite and looped crwaling of a demo page with all sort of modules of our cms https://en.wikipedia.org/wiki/Profile-guided_optimization https://gcc.gnu.org/onlinedocs/gcc-7.4.0/gcc/Gcov-Data-Files.html The .gcda count data file is generated when a program containing object files built with the GCC -fprofile-arcs option is executed. A separate .gcda file is created for each object file compiled with this option. It contains arc transition counts, value profile counts, and some summary information.
firefox-69.0.1-3.fc30 feels much better, thank you. I think we're back at expected performance.
As mentioned, it looks like a fallout of enabled profiling, which is hardly acceptable in a production build. Isn't there anything at the build infra side to catch such issues early?
you can't catch that, when only "make prof-gen" is used and for whatever reason "make prof-use" and the profiling was skipped you have just binaries as without PGO at all and the infrastructure don't know anything about it
I mean, any "notes" or whichever metadata in the ELF? Adding Nick if he could comment from the annobin/tooling perspective, please.
Hi Jan, Sorry for not noticing this needinfo request. Annobin does indeed insert extra data in the form of ELF Notes into the firefox executable. (Assuming that it is being compiled with gcc and not clang). But this data is not loadable so it only affects the on-disk size of the binary, not the in-memory size, and it should not have any effect on the performance of firefox. Cheers Nick
Hey Nick, sorry for not being clear enough. Especially now in the light of: https://fedoraproject.org/wiki/Changes/ANNOBIN-used-by-bodhi I think it might be very reasonable to be super-careful about profiling being accidentally enabled through the build switches. What do you think?
Hi Jan, OK, so yes and no: * Yes - package maintainers should be careful not to enable profiling in their packages unless that really want to, and they need to remember to turn it off afterwards. * No - the default build switches do not enable profiling. (Annobin is not a profiler). Package maintainers must explicitly enable profiling if they want it. * Yes - using annobin as part of the default build process does increase the size of built binaries. But it does not affect their run-time performance. Plus we have recently introduced a new step to the default build process to reduce the size of the annobin notes by merging together duplicates, and I have also made some changes to the merging algorithm so that it is noticeably more effective. We could, if desired, add a step in the annobin plugin, which records whether any profiling options have been used. Then we could add a check in the annocheck tool, which if run, would warn the user that profiling had been enabled. If this was part of the Bodhi process, then reviewers would be able to question whether enabling profiling was suitable for a production release. Is this the kind of thing that you had in mind ? Cheers Nick
> We could, if desired, add a step in the annobin plugin, which > records whether any profiling options have been used. Then we could > add a check in the annocheck tool, which if run, would warn the user > that profiling had been enabled. If this was part of the Bodhi > process, then reviewers would be able to question whether enabling > profiling was suitable for a production release. > > Is this the kind of thing that you had in mind ? This is exactly it! (there was a pretty wide grasp of the surroundings, so I was a bit worried we are not meeting initially :-) While it may not be explicitly mentioned anywhere in the guidelines (deferring just to use-predestined-fullstop wisdom), this very bug just demonstrates how impactful the accidentally (softly against the guidelines) profiling can be, and it'd be nice to include the respective check into the repertoire of "mistakes" to catch early, probably thanks to your heavy lifting on the tooling side. Thanks for considering this, Nick.
Hi Jan, OK, so I have updated annobin in rawhide (annobin-8.88-1.fc32) to add the detection and reporting of code instrumentation options. It currently detects -fsanitize=..., -finstrument-functions, -p, -pg and -fprofile-arcs. Plus any of the meta options that enable one or more of those options. Currently of course this change will have no effect on builds because annocheck is not run as part of the build process, nor as part of the Bodhi review process. But if that changes, then there is a chance that this new warning will catch other packages where profiling is still enabled. Cheers Nick
Addtitional to "-fprofile-arcs" you should also add "-fprofile-generate" and "-fprofile-use" which is the typical sequence of "make prof-gen" -> run application -> "make prof-clean" -> "make prof-use" like PHP as example
(In reply to Harald Reindl from comment #16) Hi Harald, > Addtitional to "-fprofile-arcs" you should also add "-fprofile-generate" and > "-fprofile-use" Actually there is no need for this. -fprofile-use just tells the compiler to use an already generated profile. This is probably a good thing, and it will not slow down the compiled code. -fprofile-generate does cause the compiler to generate profiling information, but it does so by internally setting the -fprofile-arcs option, and this will be caught by the newly updated annobin. Cheers Nick
You can use https://browserbench.org/Speedometer2.0/ to benchmark browser performance from UI perspective. Please compare with upstream builds or chrome/chromium. From my testing Fedora Firefox is slightly faster than upstream one and chromium, google chrome is slightly faster than Fedora Firefox and Epiphany a.k.a Gnome Web beats them all:)
Anyway, this should be already fixed.