Bug 1374038

Summary: long running sql statement on dashboard (> 90 sec)
Product: Red Hat Satellite Reporter: Chris Duryee <cduryee>
Component: Content ManagementAssignee: Chris Duryee <cduryee>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: medium Docs Contact:
Priority: high    
Version: 6.2.0CC: bbuckingham, bkearney, cduryee, chrobert, dmoessne, jcallaha, ktordeur, mmello, oshtaier, pgagne, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: Performance, PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.0.0.82-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1385761 (view as bug list) Environment:
Last Closed: 2016-11-10 08:13:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1385761, 1385841    
Attachments:
Description Flags
verification screenshot 1
none
verification screenshot 2 none

Description Chris Duryee 2016-09-07 18:50:45 UTC
Description of problem:

On a system with many applicable errata, the following sort can take upwards of 90 seconds:

Sort (cost=218,484.99..219,063.42 rows=231,373 width=879) (actual time=133,527.641..133,528.216 rows=21 loops=1)

    Sort Key: katello_errata.updated, katello_errata.id, katello_errata.uuid, katello_errata.errata_id, katello_errata.created_at, katello_errata.updated_at, katello_errata.issued, katello_errata.errata_type, katello_errata.severity, katello_errata.title, katel
    Sort Method: external merge Disk: 271928kB

This causes the dashboard to load slowly.

Version-Release number of selected component (if applicable): 6.2.1

Steps to Reproduce:
1. load postgres db with lots of applicable errata
2. view dashboard page

Actual results:

extremely long load time due to postgres having to sort on-disk instead of in memory.

Expected results:

page should load quickly

Additional info: Each row of the query is about 1K in size. This requires quite a bit of memory to sort when there are 200K+ rows. Note that it is not feasible to set work_mem high enough to handle this type of sort, since work_mem is used per query and even per subquery. The likely fix is to strip the errata text in the query and then select it back in if we still need it after obtaining the top 6 errata.

Secondary note: there were some other bugs in the past related to this query. This particular issue is for the same query, but the issue is different.

Comment 3 Bryan Kearney 2016-10-03 14:18:52 UTC
Upstream bug component is Content Management

Comment 4 Bryan Kearney 2016-10-03 16:19:32 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16763 has been resolved.

Comment 5 Chris Duryee 2016-10-05 20:14:34 UTC
workaround: disable the errata widget from the dashboard

Comment 10 jcallaha 2016-11-08 16:54:45 UTC
Verified in Satellite 6.2.4 Snap 2.

Average Dashboard load times on a system with 1k+ content hosts, each with 150 install-able errata, is 15 seconds. Removing the errata widget from the dashboard does not decrease load times. See verification screenshot 1 for load time with errata widget in place, and shot 2 for times without.

Comment 11 jcallaha 2016-11-08 16:55:07 UTC
Created attachment 1218614 [details]
verification screenshot 1

Comment 12 jcallaha 2016-11-08 16:55:25 UTC
Created attachment 1218615 [details]
verification screenshot 2

Comment 14 errata-xmlrpc 2016-11-10 08:13:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2699