Red Hat Bugzilla – Bug 1271143
Listing a single repositories with 1000 puppet modules takes 10+ minutes
Last modified: 2017-02-23 14:44:29 EST
Description of problem: Listing a repositories with almost 1000 puppet modules takes 10 minutes. The flattening the shows that 80% of the returned data are the repositories results. But this is for the requested query to list a specified repository superfluous data. [crash] root@li-lc-1578:~# time curl -K /opt/hoici/etc/sat6/curl-hoici.conf "-HContent-Type: application/json" "-d{\"per_page\":9999}" -XGET https://localhost/katello/api/v2/repositories/158/puppet_modules > p.json real 10m30.165s user 0m0.271s sys 0m0.080s [crash] root@li-lc-1578:~# ls -l p.json -rw-r--r--. 1 root root 1615271 Oct 13 08:39 p.json [crash] root@li-lc-1578:~# hoi-json p.json --get results.id | wc -l 988 [crash] root@li-lc-1578:~# hoi-json p.json --get ALL | wc -l 57130 [crash] root@li-lc-1578:~# hoi-json p.json --get ALL | grep -c '\.repositories\.' 45770 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create a puppet repository with 1000 puppet modules 2. List puppet repository using the API 3. Actual results: Slow repsonse Response contains redundant repositories information Expected results: Response <10seconds Response does not include redundant repository information Additional info:
Created attachment 1082304 [details] json response of repository puppet modules
From the Logging it can be seen that 99% of the time is spend in the rendering of the view. 2015-10-13 08:29:10 [I] Processing by Katello::Api::V2::PuppetModulesController#index as */* 2015-10-13 08:29:10 [I] Parameters: {"per_page"=>9999, "api_version"=>"v2", "repository_id"=>"158", "puppet_module"=>{"per_page"=>9999}} 2015-10-13 08:29:10 [I] Authorized user hoici(hoici ) ... 2015-10-13 08:39:38 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.67/app/views/katello/api/v2/puppet_modules/index.json.rabl within katello/api/v2/layouts/collection (616511.4ms) 2015-10-13 08:39:39 [I] Completed 200 OK in 629043ms (Views: 615725.0ms | ActiveRecord: 847.5ms)
Sync plans are not used, as mentioned in the a related BZ1253802
Proposed patch: diff --git a/app/views/katello/api/v2/puppet_modules/show.json.rabl b/app/views/katello/api/v2/puppet_modules/show.json.rabl index 2c4fde0..e72285b 100644 --- a/app/views/katello/api/v2/puppet_modules/show.json.rabl +++ b/app/views/katello/api/v2/puppet_modules/show.json.rabl @@ -15,5 +15,5 @@ attributes :tag_list attributes :repoids child :repositories => :repositories do |_repository| - extends 'katello/api/v2/repositories/show' + attributes :id, :name end
Patch confirmed to work, below is the result with the patch applied: [crash] root@li-lc-1578:~# time curl -K /opt/hoici/etc/sat6/curl-hoici.conf "-HContent-Type: application/json" "-d{\"per_page\":9999}" -XGET https://localhost/katello/api/v2/repositories/158/puppet_modules > p.json real 0m21.139s user 0m0.215s sys 0m0.150s [crash] root@li-lc-1578:~# ls -l p.json -rw-r--r--. 1 root root 395472 Oct 13 11:24 p.json [crash] root@li-lc-1578:~# hoi-json p.json --get ALL | wc -l 13444 [crash] root@li-lc-1578:~# hoi-json p.json --get ALL | grep -c '\.repositories\.' 1990 2015-10-13 11:24:26 [I] Processing by Katello::Api::V2::PuppetModulesController#index as */* 2015-10-13 11:24:26 [I] Parameters: {"per_page"=>9999, "api_version"=>"v2", "repository_id"=>"158", "puppet_module"=>{"per_page"=>9999}} 2015-10-13 11:24:27 [I] Authorized user hoici(hoici ) 2015-10-13 11:24:45 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.67/app/views/katello/api/v2/puppet_modules/index.json.rabl within katello/api/v2/layouts/collection (6709.1ms) 2015-10-13 11:24:46 [I] Completed 200 OK in 19596ms (Views: 7322.7ms | ActiveRecord: 61.2ms)
Verified. Used 3,684 puppet modules in a single repository. Response time was 730ms in RHEL 7.1 and 820ms in RHEL 6.7 GET - https://cloud-qe-11.idmqe.lab.eng.bos.redhat.com/katello/api/repositories/4/puppet_modules total modules: 3,684. Response time: 730ms HOSTNAME: cloud-qe-11.idmqe.lab.eng.bos.redhat.com OS: redhat RELEASE: Red Hat Enterprise Linux Server release 7.1 (Maipo) FOREMAN: 1.7.2 SATELLITE: 6.1.4 RUBY: ruby 2.0.0p598 (2014-11-13) [x86_64-linux] PUPPET: 3.6.2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:2474