Bug 1419139

Summary: packages page is slow
Product: Red Hat Satellite Reporter: Chris Duryee <cduryee>
Component: API - ContentAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: medium Docs Contact:
Priority: high    
Version: 6.2.7CC: abalakht, andrew.schofield, bbuckingham, bkearney, cduryee, egolov, jcallaha, jomitsch, pwaghmar, rplevka, sghai
Target Milestone: UnspecifiedKeywords: Performance, PrioBumpField, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-katello-3.0.0.153-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1480355 (view as bug list) Environment:
Last Closed: 2017-09-25 18:59:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1465146, 1530699    
Attachments:
Description Flags
long load
none
average load none

Description Chris Duryee 2017-02-03 16:52:58 UTC 
Description of problem:

If the katello install has a large number of packages, the packages page can be very slow (over 18 seconds). In particular, this API call takes quite some time to return:

https://localhost/katello/api/v2/packages?organization_id=1&page=1&paged=true&search=

Ideally, the page would load in less than 5 seconds.


Version-Release number of selected component (if applicable): 6.2.7


How reproducible:


Steps to Reproduce:
1. sync a bunch of repos containing numerous packages
2. go to content -> packages page
3. wait

Actual results: 18+ seconds of waiting


Expected results: 5 or less seconds of waiting


Additional info:

2017-02-03 11:49:07 [app] [I] Started GET "/katello/api/v2/packages?organization_id=1&page=2&paged=true&search=" for 10.10.121.39 at 2017-02-03 11:49:07 -0500
2017-02-03 11:49:07 [app] [I] Processing by Katello::Api::V2::PackagesController#index as JSON
2017-02-03 11:49:07 [app] [I]   Parameters: {"organization_id"=>"1", "page"=>"2", "paged"=>"true", "search"=>"", "api_version"=>"v2"}
2017-02-03 11:49:18 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.95/app/views/katello/api/v2/packages/index.json.rabl within katello/api/v2/layouts/collection (3513.7ms)
2017-02-03 11:49:18 [app] [I] Completed 200 OK in 10947ms (Views: 22.9ms | ActiveRecord: 10829.9ms)


the SQL is too big to paste, but the explain is at https://explain.depesz.com/s/JZG
Comment 5 Brad Buckingham 2017-02-27 18:45:52 UTC 
Created redmine issue http://projects.theforeman.org/issues/18704 from this bug
Comment 6 Satellite Program 2017-06-21 18:19:08 UTC 
Upstream bug assigned to jsherril
Comment 7 Satellite Program 2017-06-26 20:17:18 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18704 has been resolved.
Comment 10 jcallaha 2017-08-28 16:01:01 UTC 
Verified in Satellite 6.2.12 Snap 1.

On a system with ~45k packages, the average page load times varied between 3 and 6.5 seconds. However, there are occasional times when the page will take around 10-11 seconds to load. See attached screenshot for long and average load times.
Comment 11 jcallaha 2017-08-28 16:01:56 UTC 
Created attachment 1319141 [details]
long load
Comment 12 jcallaha 2017-08-28 16:02:23 UTC 
Created attachment 1319142 [details]
average load
Comment 18 jcallaha 2017-09-12 21:02:50 UTC 
Verified in Satellite 6.2.12 Snap 3. 

host package list working as expected, in addition to above verification.

-bash-4.2# time hammer host package list --host-id 2
[Foreman] Password for admin: 
------------------------------------------
NVRA                                      
------------------------------------------
acl-2.2.51-12.el7.x86_64                  
audit-libs-2.4.1-5.el7.x86_64             
basesystem-10.0-7.el7.noarch              
bash-4.2.46-20.el7_2.x86_64               
binutils-2.23.52.0.1-55.el7.x86_64        
bzip2-libs-1.0.6-13.el7.x86_64            
ca-certificates-2015.2.6-70.1.el7_2.noarch
chkconfig-1.3.61-5.el7_2.1.x86_64         
coreutils-8.22-15.el7_2.1.x86_64          
cpio-2.11-24.el7.x86_64                   
cracklib-2.9.0-11.el7.x86_64              
cracklib-dicts-2.9.0-11.el7.x86_64        
cryptsetup-libs-1.6.7-1.el7.x86_64        
curl-7.29.0-25.el7.x86_64                 
cyrus-sasl-lib-2.1.26-20.el7_2.x86_64     
dbus-1.6.12-14.el7_2.x86_64               
dbus-glib-0.100-7.el7.x86_64              
dbus-libs-1.6.12-14.el7_2.x86_64          
dbus-python-1.1.1-9.el7.x86_64            
device-mapper-1.02.107-5.el7_2.5.x86_64   
------------------------------------------
Page 1 of 9 (use --page and --per-page for navigation)

real	0m4.906s
user	0m1.870s
sys	0m0.225s


additionally

[root@ibm-x3550m3-06 ~]# time hammer -p changeme package list --organization-id 1
...
23136 | zziplib-0.13.62-5.el7.i686.rpm                                                  
13106 | zziplib-devel-0.13.62-5.el7.x86_64.rpm                                          
12710 | zziplib-devel-0.13.62-5.el7.i686.rpm                                            
9240  | zziplib-utils-0.13.62-5.el7.x86_64.rpm                                          
------|----------------------------------------

real	2m29.837s
user	0m34.504s
sys	0m1.717s
[root@ibm-x3550m3-06 ~]# 
[root@ibm-x3550m3-06 ~]# time hammer -p changeme package list --organization-id 1 > packages.txt

real	2m15.434s
user	0m27.393s
sys	0m0.987s
[root@ibm-x3550m3-06 ~]# wc -l packages.txt 
39022 packages.txt
Comment 19 Sachin Ghai 2017-09-25 11:25:19 UTC 
*** Bug 1489403 has been marked as a duplicate of this bug. ***
Comment 21 errata-xmlrpc 2017-09-25 18:59:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2803