Bug 1250083

Description of problem:
The REST timeout of 60 seconds to retrieve the list of puppet classes from foreman smart proxy can be too small when the Sat6 is under load.

But initial response time of a request is without caching is 77 seconds, this is still too slow that the REST from Katello call times out after 60 seconds.
A second call to the smart proxy takes only 3 seconds, so the caching is working for that case:

# zgrep hg_crash_ foreman-proxy/proxy.log-20150804.gz
10.92.14.128 - - [03/Aug/2015 12:54:19] "GET /puppet/environments/KT_Hilti_Library_hg_crash_18/classes HTTP/1.1" 200 11417 76.6783
10.92.14.128 - - [03/Aug/2015 12:56:51] "GET /puppet/environments/KT_Hilti_Library_hg_crash_18/classes HTTP/1.1" 200 11417 3.5445


The correspond log from katello, that shows it times out 16 seconds before the smart proxy finished processing the request, that is matches nicely with a REST timeout setting of 60 seconds:

2015-08-03 12:54:03 [E] ERF12-4115 [ProxyAPI::ProxyException]: Impossible d'obtenir les classes Puppet pour KT_Hilti_Library_hg_crash_18 ([RestClient::RequestTimeout]: Request Timeout) pour le proxy https://li-lc-1578.hag.hilti.com:9090/puppet (ProxyAPI::ProxyException)
/usr/share/foreman/lib/proxy_api/puppet.rb:27:in `rescue in classes'
/usr/share/foreman/lib/proxy_api/puppet.rb:21:in `classes'
/usr/share/foreman/app/services/puppet_class_importer.rb:155:in `actual_classes'
/usr/share/foreman/app/services/puppet_class_importer.rb:78:in `new_classes_for'
/usr/share/foreman/app/services/puppet_class_importer.rb:39:in `changes'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/services/katello/puppet_class_importer_extensions.rb:22:in `update_environment'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/katello/foreman.rb:36:in `update_puppet_environment'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/katello/foreman/content_update.rb:32:in `finalize'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:503:in `block (2 levels) in execute_finalize'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass'


Testing from the commandline using curl with a clean caches and heavy disk IO it can be reproduced:

# <reboot server to have no caches>

# <start repository sync with heavy disk io>

# nproc && uptime
8
 13:19:17 up 36 min,  3 users,  load average: 9.07, 9.35, 5.80

# puppetenv=$(ls -1 /etc/puppet/environments | grep -v example | head -n1)

# puppetenv=$(ls -1 /etc/puppet/environments | grep -v example | head -n1)                                                                                  [crash] root@li-lc-1578:~# time curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem -k -H "Accept: application/json" https://$(hostname -f):9090/puppet/environments/$puppetenv/classes > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11417  100 11417    0     0    157      0  0:01:12  0:01:12 --:--:--  2814

real    1m12.527s
user    0m0.191s
sys     0m0.056s

# time find /etc/puppet/environments/$puppetenv/modules -name '*.pp' | while read f; do grep '^class [[:alpha:]]' $f; done | wc -l
77

real    0m1.229s
user    0m0.160s
sys     0m0.380s


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create Content View A
1. Add 50 puppet modules with 100 classes
2. Start a Load using Repository Sync:
2a. Select all repositories
2b. Click Sync now Redhat channel sync 
3. Publish Content View A
4. Watch the response time of smart-proxy


Actual results:
Timeout when in 

Expected results:


Additional info: