Bug 1067680 - Passenger instances doesn't seem to scale up to meet the demand of parallel requests.
Summary: Passenger instances doesn't seem to scale up to meet the demand of parallel r...
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Abhishek Gupta
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-02-20 20:43 UTC by Peter Ruan
Modified: 2015-05-15 00:25 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-03-12 13:37:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
broker.log (13.46 KB, text/x-log)
2014-02-20 20:43 UTC, Peter Ruan
no flags Details
mcollective.log (616.66 KB, text/x-log)
2014-02-20 20:44 UTC, Peter Ruan
no flags Details
condensed mcollective.log (8.07 KB, text/x-log)
2014-02-20 20:45 UTC, Peter Ruan
no flags Details

Description Peter Ruan 2014-02-20 20:43:13 UTC
Created attachment 865679 [details]
broker.log

Description of problem:
  If a user tries to create multiple apps in parallel, the Passenger count seems to stuck at the what /var/www/openshift/broker/httpd/broker.conf's PassengerMinInstances (default to 2).  As a result of this, the POST requests are queued up in a set of 2's until the previous queued-up requests have finished executing.

Version-Release number of selected component (if applicable):
devenv/origin/ose

How reproducible:
always.

Steps to Reproduce:
run the following ruby script...changing the parameter as necessary to fit your setup.

#!/usr/bin/env ruby
thread_list = []
(1..30).each  { |n|
  thread_list << Thread.new {
    if (n % 2) == 0
      cmd = "curl -k -H 'Accept: application/xml' --user demo:changeme  https://10.14.16.227/broker/rest/domain/origin/applications   -d name=app#{n} -d cartridges[]=php-5.3 -X POST"
    else
      cmd = "curl -k -H 'Accept: application/xml' --user demo:changeme  https://10.14.16.227/broker/rest/domain/origin/applications -X GET"
    end
    system(cmd)
  }
}
thread_list.each {|x| x.join}



Actual results:
the HTTP requests are not executed in parallel because the Passenger instances are not scaling up to meet the demands.

Expected results:
HTTP requests are executed in parallel

Additional info:

Comment 1 Peter Ruan 2014-02-20 20:44:41 UTC
Created attachment 865680 [details]
mcollective.log

Comment 2 Peter Ruan 2014-02-20 20:45:58 UTC
Created attachment 865682 [details]
condensed mcollective.log

Comment 3 Dan McPherson 2014-02-26 19:24:48 UTC
Abhishek, was there something else you were going to do here?  Can close the remainder of the bug on the scheduler card I think.

Comment 4 Dan McPherson 2014-03-12 13:37:02 UTC
The min instances have been increases to 4 and the live time has been raised to 10 mins.  If you run real load tests the additional threads will kick in and meet the demand.  While this isn't perfect, it's the best we are going to be able to do for now.


Note You need to log in before you can comment on or make changes to this bug.