Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1374038 - long running sql statement on dashboard (> 90 sec)
long running sql statement on dashboard (> 90 sec)
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Content Management (Show other bugs)
6.2.0
Unspecified Unspecified
high Severity medium (vote)
: 6.2.4
: Unused
Assigned To: Chris Duryee
jcallaha
: Performance, PrioBumpGSS, Triaged
Depends On:
Blocks: CEE_Sat6_Top_BZs/GSS_Sat6_Top_Bugs 1385761
  Show dependency treegraph
 
Reported: 2016-09-07 14:50 EDT by Chris Duryee
Modified: 2017-03-15 08:47 EDT (History)
12 users (show)

See Also:
Fixed In Version: tfm-rubygem-katello-3.0.0.82-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1385761 (view as bug list)
Environment:
Last Closed: 2016-11-10 03:13:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
verification screenshot 1 (152.57 KB, image/png)
2016-11-08 11:55 EST, jcallaha
no flags Details
verification screenshot 2 (172.84 KB, image/png)
2016-11-08 11:55 EST, jcallaha
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2696191 None None None 2016-10-12 16:19 EDT
Foreman Issue Tracker 16763 None None None 2016-10-03 08:09 EDT
Red Hat Product Errata RHBA-2016:2699 normal SHIPPED_LIVE Satellite 6.2.4 Async Bug Release 2016-11-10 08:12:22 EST

  None (edit)
Description Chris Duryee 2016-09-07 14:50:45 EDT
Description of problem:

On a system with many applicable errata, the following sort can take upwards of 90 seconds:

Sort (cost=218,484.99..219,063.42 rows=231,373 width=879) (actual time=133,527.641..133,528.216 rows=21 loops=1)

    Sort Key: katello_errata.updated, katello_errata.id, katello_errata.uuid, katello_errata.errata_id, katello_errata.created_at, katello_errata.updated_at, katello_errata.issued, katello_errata.errata_type, katello_errata.severity, katello_errata.title, katel
    Sort Method: external merge Disk: 271928kB

This causes the dashboard to load slowly.

Version-Release number of selected component (if applicable): 6.2.1

Steps to Reproduce:
1. load postgres db with lots of applicable errata
2. view dashboard page

Actual results:

extremely long load time due to postgres having to sort on-disk instead of in memory.

Expected results:

page should load quickly

Additional info: Each row of the query is about 1K in size. This requires quite a bit of memory to sort when there are 200K+ rows. Note that it is not feasible to set work_mem high enough to handle this type of sort, since work_mem is used per query and even per subquery. The likely fix is to strip the errata text in the query and then select it back in if we still need it after obtaining the top 6 errata.

Secondary note: there were some other bugs in the past related to this query. This particular issue is for the same query, but the issue is different.
Comment 3 Bryan Kearney 2016-10-03 10:18:52 EDT
Upstream bug component is Content Management
Comment 4 Bryan Kearney 2016-10-03 12:19:32 EDT
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16763 has been resolved.
Comment 5 Chris Duryee 2016-10-05 16:14:34 EDT
workaround: disable the errata widget from the dashboard
Comment 10 jcallaha 2016-11-08 11:54:45 EST
Verified in Satellite 6.2.4 Snap 2.

Average Dashboard load times on a system with 1k+ content hosts, each with 150 install-able errata, is 15 seconds. Removing the errata widget from the dashboard does not decrease load times. See verification screenshot 1 for load time with errata widget in place, and shot 2 for times without.
Comment 11 jcallaha 2016-11-08 11:55 EST
Created attachment 1218614 [details]
verification screenshot 1
Comment 12 jcallaha 2016-11-08 11:55 EST
Created attachment 1218615 [details]
verification screenshot 2
Comment 14 errata-xmlrpc 2016-11-10 03:13:54 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2699

Note You need to log in before you can comment on or make changes to this bug.