Bug 1393053

Summary: Bugzilla performance in Brno
Product: [Community] Bugzilla Reporter: Alois Mahdal <amahdal>
Component: PerformanceAssignee: PnT DevOps Devs <hss-ied-bugs>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.4CC: amahdal, azelinka, bcodding, huiwang, qgong
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-21 03:32:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1775514    
Bug Blocks:    
Attachments:
Description Flags
2year visit stats for amahdal
none
opening a query (second+ time today)
none
opening another query (first time today)
none
opening a "random" bug none

Description Alois Mahdal 2016-11-08 18:07:52 UTC
Disclaimer
==========

Information below is not to be considered a metric in any way.  It's based purely on my (amahdal) own usage and experience, "confirmed" by talking to fellow Red Hatters around.  I'm not aware of any proper research on this topic.


My workflow
===========

As QE engineer, my daily usage patterns of internal web UI tools could be described as:

 *  1/3 Red Hat Bugzilla
 *  1/3 Beaker
 *  1/3 everything else

I believe that the importance of Bugzilla in daily workflow will be similar for most of QE engineers (at Platform QE, at least, but also for many other engineers).


Performance
===========

But unfortunately at least here in Brno we still experience load times of 10-15 seconds as "norm", ranking this Bugzilla instance by far as one of slowest tools used in daily workflow.  Needless to say, this can be frustrating at times, and also increases potential for disruption (even 5s can be a lot of time these days).

Now, it's important to say that there have been significant improvements in Bugzilla performance in last few years.  I'm aware of and appreciating that effort, but I still believe Red Hat could do much better at this.

Comment 1 Alois Mahdal 2016-11-08 18:10:16 UTC
Created attachment 1218640 [details]
2year visit stats for amahdal

Just for illustration, my own visit stats collected for last 2 years or so.

Comment 2 Jeff Fearn 🐞 2016-11-08 23:47:38 UTC
Could you use your browsers developer tools to get a time line of a page load or two and attach the timings?

In firefox you can save the stats as HAR and attach it.

Comment 3 Jeff Fearn 🐞 2016-11-09 00:05:35 UTC
I found a site that allowed me to run a page load test for this bug from Stockholm, it's about the same time as  it takes for me here in Brisbane, 5.95s, for a fresh load that has to fetch all the JS and CSS files.

Not sure this link is reusable 

https://tools.pingdom.com/#!/cRAFPX/﷒0﷓

Some recommendations from the test:

f 8 	Remove query strings from static resources

Resources with a "?" in the URL are not cached by some proxy caching servers. Remove the query string and encode the parameters into the URL for the following resources:

    https://bugzilla.redhat.com/extensions/BayotBase/web/css/base.css?1447136797
    https://bugzilla.redhat.com/extensions/BayotBase/web/css/jquery-ui-1.8.18.custom.css?1447136797
    https://bugzilla.redhat.com/extensions/BayotBase/web/js/Base.js?1447136797
    https://bugzilla.redhat.com/extensions/BayotBase/web/js/bayot.util.js?1457925561
    https://bugzilla.redhat.com/extensions/BayotBase/web/js/jquery-ui-1.8.18.custom.min.js?1447136797
    https://bugzilla.redhat.com/extensions/BayotBase/web/js/jquery.cookie.js?1447136797
    https://bugzilla.redhat.com/extensions/BayotBase/web/js/jquery.jsonrpc.js?1447136797
    https://bugzilla.redhat.com/extensions/FontAwesome/web/css/font-awesome.min.css?1447136797
    https://bugzilla.redhat.com/extensions/SelectizeJS/web/css/SelectizeJS.css?1457925561
    https://bugzilla.redhat.com/extensions/SelectizeJS/web/css/selectize.bootstrap3.css?1447136797
    https://bugzilla.redhat.com/extensions/SelectizeJS/web/js/SelectizeJS.js?1460334824
    https://bugzilla.redhat.com/extensions/SelectizeJS/web/js/standalone/selectize.js?1457925561
    https://bugzilla.redhat.com/extensions/VCS/web/style.css?1447136797
    https://bugzilla.redhat.com/extensions/Voting/web/style.css?1457925561
    https://bugzilla.redhat.com/js/comments.js?1457925561
    https://bugzilla.redhat.com/js/field.js?1457925561
    https://bugzilla.redhat.com/js/global.js?1457925561
    https://bugzilla.redhat.com/js/productform.js?1457925561
    https://bugzilla.redhat.com/js/show_bug.js?1457925561
    https://bugzilla.redhat.com/js/util.js?1457925561
    https://bugzilla.redhat.com/js/yui/assets/skins/sam/autocomplete.css?1447136797
    https://bugzilla.redhat.com/js/yui/assets/skins/sam/calendar.css?1447136797
    https://bugzilla.redhat.com/js/yui/autocomplete/autocomplete-min.js?1447136797
    https://bugzilla.redhat.com/js/yui/calendar/calendar-min.js?1447136797
    https://bugzilla.redhat.com/js/yui/connection/connection-min.js?1447136797
    https://bugzilla.redhat.com/js/yui/cookie/cookie-min.js?1447136797
    https://bugzilla.redhat.com/js/yui/datasource/datasource-min.js?1447136797
    https://bugzilla.redhat.com/js/yui/json/json-min.js?1447136797
    https://bugzilla.redhat.com/js/yui/yahoo-dom-event/yahoo-dom-event.js?1447136797
    https://bugzilla.redhat.com/skins/contrib/RedHat/global.css?1457925561
    https://bugzilla.redhat.com/skins/contrib/RedHat/show_bug.css?1457925561
    https://bugzilla.redhat.com/skins/standard/global.css?1457925561
    https://bugzilla.redhat.com/skins/standard/show_bug.css?1457925561


See: https://developers.google.com/speed/docs/insights/rules

     
c 70 	Serve static content from a cookieless domain
Serve the following static resources from a domain that doesn't set cookies:

-- All the above files listed again. --

Everything else was "A 100"

The number after the '?' is the mtime of the file.

Comment 4 Alois Mahdal 2016-11-28 14:15:36 UTC
Today I tried loading several pages but indeed, the load times are slightly under 6s.  (Frankly it still seems a lot, although it's still better than my 10-15 guess.)

However, acessing an unknown bug or a query that I haven't opened for a long time, the times get much worse.  For example, today, opening my most frequently used query, "bugs by me", took roughly 20s, but re-loading it took about 5.

Similarly I tried to open "random" bug, and it took ~10s to load, then reloading was more like 4-6s.  Apparently some caching influences this heavily, which makes it hard to reproduce consistently.

Sadly I did not catch the timeline of the worst loads, and subsequent loads get better---even in other browsers.  I'm attaching at least few  HAR files.  Hopefully they will be helpful.

Comment 5 Alois Mahdal 2016-11-28 14:16:44 UTC
Created attachment 1225301 [details]
opening a query (second+ time today)

Comment 6 Alois Mahdal 2016-11-28 14:17:29 UTC
Created attachment 1225302 [details]
opening another query (first time today)

Comment 7 Alois Mahdal 2016-11-28 14:18:15 UTC
Created attachment 1225304 [details]
opening a "random" bug

Comment 8 Jeff Fearn 🐞 2020-04-07 01:57:38 UTC
Hi Alois, we've shipped 4 bugs recently that should help with this, have you noticed any change in Brno so far this week?

Bug 1819032 Combine CSS files as much as possible
Bug 1819033 Combine JS files as much as possible
Bug 1819229 Increase doPoll timeout dramatically
Bug 1775514 Bug edit page regenerates a lot of data it could cache

Comment 9 Alois Mahdal 2020-04-07 07:13:29 UTC
Yup, it's much better, Jeff!

 *  "bugs by me" takes 4-5s (instead of 20s),
 *  (but other query with, much shorter resiult takkes about 2s),
 *  "random bug" takes 2.5s (instead of 5s).

Great work!

~

Here are some engineers working from homes, happy about BZ performance.

   /=================================\
  //   \__   ||   \__   ||   o<<     \\
  ||         ||         ||           ||
  ||   \o/   ||   \o/   ||   \o/     ||
  ||    |    ||    |    ||    |      ||
  ||   / \   ||   / \   ||   / \ \__ ||
  '==================================='

(The one on the right is a happy father, he's just about to find out about it..)

Comment 10 Jeff Fearn 🐞 2020-04-21 03:32:29 UTC
Hi, this seems to be pretty stable so I'm going to close this bug.

We are looking to see if we can make further improvements on Bug 1812752, but we need to reboot that effort after the recent changes.