Bug 1271143 - Listing a single repositories with 1000 puppet modules takes 10+ minutes
Listing a single repositories with 1000 puppet modules takes 10+ minutes
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Content Management (Show other bugs)
6.1.2
All Linux
high Severity high (vote)
: 6.1.4
: --
Assigned To: Justin Sherrill
jcallaha
: Triaged
Depends On:
Blocks: 1122832
  Show dependency treegraph
 
Reported: 2015-10-13 04:59 EDT by Peter Vreman
Modified: 2017-02-23 14:44 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 10:57:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
json response of repository puppet modules (1.54 MB, text/plain)
2015-10-13 05:01 EDT, Peter Vreman
no flags Details

  None (edit)
Description Peter Vreman 2015-10-13 04:59:39 EDT
Description of problem:
Listing a repositories with almost 1000 puppet modules takes 10 minutes.
The flattening the shows that 80% of the returned data are the repositories results. But this is for the requested query to list a specified repository superfluous data.

[crash] root@li-lc-1578:~# time curl -K /opt/hoici/etc/sat6/curl-hoici.conf "-HContent-Type: application/json" "-d{\"per_page\":9999}" -XGET https://localhost/katello/api/v2/repositories/158/puppet_modules > p.json

real    10m30.165s
user    0m0.271s
sys     0m0.080s

[crash] root@li-lc-1578:~# ls -l p.json
-rw-r--r--. 1 root root 1615271 Oct 13 08:39 p.json

[crash] root@li-lc-1578:~# hoi-json p.json --get results.id | wc -l
988

[crash] root@li-lc-1578:~# hoi-json p.json --get ALL | wc -l
57130

[crash] root@li-lc-1578:~# hoi-json p.json --get ALL | grep -c '\.repositories\.'
45770


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create a puppet repository with 1000 puppet modules
2. List puppet repository using the API
3.

Actual results:
Slow repsonse
Response contains redundant repositories information

Expected results:
Response <10seconds
Response does not include redundant repository information

Additional info:
Comment 1 Peter Vreman 2015-10-13 05:01 EDT
Created attachment 1082304 [details]
json response of repository puppet modules
Comment 2 Peter Vreman 2015-10-13 05:26:02 EDT
From the Logging it can be seen that 99% of the time is spend in the rendering of the view.

2015-10-13 08:29:10 [I] Processing by Katello::Api::V2::PuppetModulesController#index as */*
2015-10-13 08:29:10 [I]   Parameters: {"per_page"=>9999, "api_version"=>"v2", "repository_id"=>"158", "puppet_module"=>{"per_page"=>9999}}
2015-10-13 08:29:10 [I] Authorized user hoici(hoici )
...
2015-10-13 08:39:38 [I]   Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.67/app/views/katello/api/v2/puppet_modules/index.json.rabl within
katello/api/v2/layouts/collection (616511.4ms)
2015-10-13 08:39:39 [I] Completed 200 OK in 629043ms (Views: 615725.0ms | ActiveRecord: 847.5ms)
Comment 3 Peter Vreman 2015-10-13 05:46:25 EDT
Sync plans are not used, as mentioned in the a related BZ1253802
Comment 4 Justin Sherrill 2015-10-13 05:52:45 EDT
Proposed patch:

diff --git a/app/views/katello/api/v2/puppet_modules/show.json.rabl b/app/views/katello/api/v2/puppet_modules/show.json.rabl
index 2c4fde0..e72285b 100644
--- a/app/views/katello/api/v2/puppet_modules/show.json.rabl
+++ b/app/views/katello/api/v2/puppet_modules/show.json.rabl
@@ -15,5 +15,5 @@ attributes :tag_list
 attributes :repoids
 
 child :repositories => :repositories do |_repository|
-  extends 'katello/api/v2/repositories/show'
+  attributes :id, :name
 end
Comment 6 Peter Vreman 2015-10-13 07:29:53 EDT
Patch confirmed to work, below is the result with the patch applied:


[crash] root@li-lc-1578:~# time curl -K /opt/hoici/etc/sat6/curl-hoici.conf "-HContent-Type: application/json" "-d{\"per_page\":9999}" -XGET https://localhost/katello/api/v2/repositories/158/puppet_modules > p.json

real    0m21.139s
user    0m0.215s
sys     0m0.150s

[crash] root@li-lc-1578:~# ls -l p.json
-rw-r--r--. 1 root root 395472 Oct 13 11:24 p.json

[crash] root@li-lc-1578:~# hoi-json p.json --get ALL | wc -l
13444

[crash] root@li-lc-1578:~# hoi-json p.json --get ALL | grep -c '\.repositories\.'
1990

2015-10-13 11:24:26 [I] Processing by Katello::Api::V2::PuppetModulesController#index as */*
2015-10-13 11:24:26 [I]   Parameters: {"per_page"=>9999, "api_version"=>"v2", "repository_id"=>"158", "puppet_module"=>{"per_page"=>9999}}
2015-10-13 11:24:27 [I] Authorized user hoici(hoici )
2015-10-13 11:24:45 [I]   Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.67/app/views/katello/api/v2/puppet_modules/index.json.rabl within katello/api/v2/layouts/collection (6709.1ms)
2015-10-13 11:24:46 [I] Completed 200 OK in 19596ms (Views: 7322.7ms | ActiveRecord: 61.2ms)
Comment 9 jcallaha 2015-11-04 12:44:04 EST
Verified. Used 3,684 puppet modules in a single repository. Response time was 730ms in RHEL 7.1 and 820ms in RHEL 6.7

GET - https://cloud-qe-11.idmqe.lab.eng.bos.redhat.com/katello/api/repositories/4/puppet_modules

total modules: 3,684.
Response time: 730ms

 HOSTNAME: cloud-qe-11.idmqe.lab.eng.bos.redhat.com
       OS: redhat
  RELEASE: Red Hat Enterprise Linux Server release 7.1 (Maipo)
  FOREMAN: 1.7.2
SATELLITE: 6.1.4
     RUBY: ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
   PUPPET: 3.6.2
Comment 14 errata-xmlrpc 2015-11-19 10:57:13 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:2474

Note You need to log in before you can comment on or make changes to this bug.