Bug 995637

Summary: mcollective performance problem when the number of parallel requests = 20.
Product: OpenShift Online Reporter: Peter Ruan <pruan>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED NOTABUG QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: unspecified    
Version: 2.xCC: dmcphers
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-08 00:46:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
mcollective.log
none
development.log none

Description Peter Ruan 2013-08-09 23:01:21 UTC
Description of problem:
  I'm doing parallel cucumber tests against a single user/domain.  I started with 10 threads running 10 different independent scenarios (creating 10 different type of apps and check if I can control it (start/stop/reload, and etc).  This runs fine.  However when I up the threads/test counts to 20, I would get between 3-7 failures out of the 20 apps, with mostly error message like below...  This was run against the latest devenv image.  When I run the same set of tests in PROD, it beaks similarly on 10 threads only. 


[15:25:15] INFO> Shell Command: rhc app create rxulwc jbossas-7 -l pruan -p 'vostok08' --insecure --server ec2-54-211-9-30.compute-1.amazonaws.com
      Application Options
      -------------------
        Namespace:  pruan08
        Cartridges: jbossas-7
        Gear Size:  default
        Scaling:    no
      
      Creating application 'rxulwc' ... An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift server 'https://ec2-54-211-9-30.compute-1.amazonaws.com/broker/rest/domains/pruan08/applications'.
      [15:30:23] INFO> Exit Status: 1
      [15:30:23] WARN> Expected response not found: Your application 'rxulwc' is now available


[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Got Node Unavailable Exception (pid:32355)
^[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Reference ID: 84263f4f4b6e86f1ba84e059b14901c4 - No nodes available.



Version-Release number of selected component (if applicable):
devenv

How reproducible:
always

Steps to Reproduce:
1. create  10/20 apps of different types in parallel against a devenv.
2.
3.

Actual results:
Failing Scenarios:
cucumber features/demo19.feature:2 # Scenario: Demo19
cucumber features/demo18.feature:2 # Scenario: Demo18
cucumber features/demo6.feature:2 # Scenario: Demo6
cucumber features/demo7.feature:2 # Scenario: Demo7
cucumber features/demo8.feature:2 # Scenario: Demo8
cucumber features/demo12.feature:2 # Scenario: Demo12
cucumber features/demo9.feature:2 # Scenario: Demo9

20 scenarios (7 failed, 13 passed)
500 steps (7 failed, 161 skipped, 332 passed)

Took 585.512743205 seconds


Expected results:
be able to create & control independent apps in parallel.

Additional info:

Comment 1 Peter Ruan 2013-08-09 23:05:30 UTC
Created attachment 785032 [details]
mcollective.log

Comment 2 Peter Ruan 2013-08-09 23:07:40 UTC
Created attachment 785034 [details]
development.log