Bug 1067680

Summary: Passenger instances doesn't seem to scale up to meet the demand of parallel requests.
Product: OpenShift Online Reporter: Peter Ruan <pruan>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CANTFIX QA Contact: libra bugs <libra-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.xCC: dmcphers
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-12 13:37:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
broker.log
none
mcollective.log
none
condensed mcollective.log none

Description Peter Ruan 2014-02-20 20:43:13 UTC
Created attachment 865679 [details]
broker.log

Description of problem:
  If a user tries to create multiple apps in parallel, the Passenger count seems to stuck at the what /var/www/openshift/broker/httpd/broker.conf's PassengerMinInstances (default to 2).  As a result of this, the POST requests are queued up in a set of 2's until the previous queued-up requests have finished executing.

Version-Release number of selected component (if applicable):
devenv/origin/ose

How reproducible:
always.

Steps to Reproduce:
run the following ruby script...changing the parameter as necessary to fit your setup.

#!/usr/bin/env ruby
thread_list = []
(1..30).each  { |n|
  thread_list << Thread.new {
    if (n % 2) == 0
      cmd = "curl -k -H 'Accept: application/xml' --user demo:changeme  https://10.14.16.227/broker/rest/domain/origin/applications   -d name=app#{n} -d cartridges[]=php-5.3 -X POST"
    else
      cmd = "curl -k -H 'Accept: application/xml' --user demo:changeme  https://10.14.16.227/broker/rest/domain/origin/applications -X GET"
    end
    system(cmd)
  }
}
thread_list.each {|x| x.join}



Actual results:
the HTTP requests are not executed in parallel because the Passenger instances are not scaling up to meet the demands.

Expected results:
HTTP requests are executed in parallel

Additional info:

Comment 1 Peter Ruan 2014-02-20 20:44:41 UTC
Created attachment 865680 [details]
mcollective.log

Comment 2 Peter Ruan 2014-02-20 20:45:58 UTC
Created attachment 865682 [details]
condensed mcollective.log

Comment 3 Dan McPherson 2014-02-26 19:24:48 UTC
Abhishek, was there something else you were going to do here?  Can close the remainder of the bug on the scheduler card I think.

Comment 4 Dan McPherson 2014-03-12 13:37:02 UTC
The min instances have been increases to 4 and the live time has been raised to 10 mins.  If you run real load tests the additional threads will kick in and meet the demand.  While this isn't perfect, it's the best we are going to be able to do for now.