Bug 1163452

Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Satellite is currently configured to handle six simulateneous requests to passenger. Additional requests will be read by apache but sit in W state until a worker is available.

Here is how I tested:

edit /usr/share/foreman/app/controllers/api/v2/users_controller.rb and alter the "index" function to be as such:

     def index
        sleep(20)
        @users = resource_scope_for_index
      end


This will make API calls for the user list take 20 seconds, which gives plenty of time to load up the workers via hammer and watch the results.

Then, make hammer calls:

hammer -u admin -p <password> user list &

Note that the first time you do this, it will take a little longer since additional workers need to spin up. Subsequent to the first run, you'll see the first X calls take 20-ish seconds, but additional calls will be serviced after the first batch drain out, or will time out.

One fix for this is to have multiple pools of workers, but I think being able to alter the PassengerMaxPoolSize setting via the installer will be sufficient, and servicing API and UI calls from the same pool vs two pools.

Upstream bug component is Provisioning

Upstream bug component is Installer

Connecting redmine issue http://projects.theforeman.org/issues/14127 from this bug

*** Bug 1329544 has been marked as a duplicate of this bug. ***

*** Bug 1333291 has been marked as a duplicate of this bug. ***

example conf addition that guarantees a certain # of workers for web UI requests


# cat /etc/httpd/conf.d/passenger_tunings.conf
# this file contains additional tunings for passenger

<IfModule mod_passenger.c>
   # PassengerMaxPoolSize is dependent on your memory and number of
   # cores. It is 6 by default.
   PassengerMaxPoolSize 10
   # PassengerMaxInstancesPerApp depends on server utilization, but 1/2 of
   # PassengerMaxPoolSize is a good starting point
   PassengerMaxInstancesPerApp 5
   PassengerMaxRequestQueueSize 250
   PassengerStatThrottleRate 120
</IfModule>

*** Bug 1356285 has been marked as a duplicate of this bug. ***

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/14023 has been resolved.

*** Bug 1360406 has been marked as a duplicate of this bug. ***

I'm still seeing quite a bit of slowdown in Satellite 6.2.7 Snap 1.

I ran 6 concurrent API host searches, as seen below, on a pretty beefy system 100k+ hosts. During the searches, an example page went from about 1.1s load times to just over 12s. Is there some kind of configuration i should be doing, or is this fix supposed to be built-in to the release?

Attaching screenshot of load times.

-bash-4.2# for i in {1..6}; do docker run -d ch-d:rhel7 curl -X GET -k -u admin:changeme -H "Content-Type: application/json" "https://rhsm-qe-1.rhq.lab.eng.bos.redhat.com/api/v2/hosts?search=name~virt&per_page=1000";done;
3cb4e34f3a1fd33724b0dd8f15d8495f7b238ac61d67fb8c78bb0a1f05f55afb
14c05b2d075b267b51da1b8b4b8dcf1341207051135efd593a761ea20e106034
905b4b2b4592af1c437a3145d056db1143c1c325d590884a6be9794e632c7a7a
c4f09546053a6025944c49958aabde321a26dd9e38e87d54b1b0d02a136bbe4b
0dfc6b9662a45937196ed0d527ebb674685998d2c911b67742276535e66f1370
64914c57b7293b6567f92dcc43fe6271d98599161cd28f2df75b0853e08d4743

{
  "total": 101906,
  "subtotal": 99609,
  "page": 1,
  "per_page": 1000,
  "search": "name~virt",
  "sort": {
    "by": null,
    "order": null
  },
  "results": [
....

Created attachment 1241412 [details]
page load time

Verified in Satellite 6.2.7 Snap 2

Followed steps in #20, scaled the number of simultaneous requests to 6, 8, 10, 12, 14, 16. For everything less than 12 requests, there was an occasional lag in page load time (6-12s) right after the requests were kicked off. However immediately after, page load times were back to normal.

When I submitted 12 or more simultaneous requests, the page load times jumped to upwards of a minute, which is expected as all the passengers were consumed. After upping the PassengerMaxPoolSize to 20, the performance returned to what was seen with less than 12 simultaneous requests.

default configuration

-bash-4.2# cat /etc/httpd/conf.modules.d/passenger_extra.conf
# The Passenger Apache module configuration file is being
# managed by Puppet and changes will be overwritten.
<IfModule mod_passenger.c>
  PassengerMaxPoolSize 12
  PassengerMaxInstancesPerApp 6
  PassengerMaxRequestQueueSize 250
  PassengerStatThrottleRate 120
</IfModule>

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0197

*** Bug 1384260 has been marked as a duplicate of this bug. ***

*** Bug 1377060 has been marked as a duplicate of this bug. ***