Bug 995637 - mcollective performance problem when the number of parallel requests = 20.
Summary: mcollective performance problem when the number of parallel requests = 20.
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: ---
Assignee: Abhishek Gupta
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-09 23:01 UTC by Peter Ruan
Modified: 2015-05-15 00:19 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-08 00:46:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
mcollective.log (1.52 MB, text/x-log)
2013-08-09 23:05 UTC, Peter Ruan
no flags Details
development.log (3.35 MB, text/x-log)
2013-08-09 23:07 UTC, Peter Ruan
no flags Details

Description Peter Ruan 2013-08-09 23:01:21 UTC
Description of problem:
  I'm doing parallel cucumber tests against a single user/domain.  I started with 10 threads running 10 different independent scenarios (creating 10 different type of apps and check if I can control it (start/stop/reload, and etc).  This runs fine.  However when I up the threads/test counts to 20, I would get between 3-7 failures out of the 20 apps, with mostly error message like below...  This was run against the latest devenv image.  When I run the same set of tests in PROD, it beaks similarly on 10 threads only. 


[15:25:15] INFO> Shell Command: rhc app create rxulwc jbossas-7 -l pruan -p 'vostok08' --insecure --server ec2-54-211-9-30.compute-1.amazonaws.com
      Application Options
      -------------------
        Namespace:  pruan08
        Cartridges: jbossas-7
        Gear Size:  default
        Scaling:    no
      
      Creating application 'rxulwc' ... An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift server 'https://ec2-54-211-9-30.compute-1.amazonaws.com/broker/rest/domains/pruan08/applications'.
      [15:30:23] INFO> Exit Status: 1
      [15:30:23] WARN> Expected response not found: Your application 'rxulwc' is now available


[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Got Node Unavailable Exception (pid:32355)
^[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Reference ID: 84263f4f4b6e86f1ba84e059b14901c4 - No nodes available.



Version-Release number of selected component (if applicable):
devenv

How reproducible:
always

Steps to Reproduce:
1. create  10/20 apps of different types in parallel against a devenv.
2.
3.

Actual results:
Failing Scenarios:
cucumber features/demo19.feature:2 # Scenario: Demo19
cucumber features/demo18.feature:2 # Scenario: Demo18
cucumber features/demo6.feature:2 # Scenario: Demo6
cucumber features/demo7.feature:2 # Scenario: Demo7
cucumber features/demo8.feature:2 # Scenario: Demo8
cucumber features/demo12.feature:2 # Scenario: Demo12
cucumber features/demo9.feature:2 # Scenario: Demo9

20 scenarios (7 failed, 13 passed)
500 steps (7 failed, 161 skipped, 332 passed)

Took 585.512743205 seconds


Expected results:
be able to create & control independent apps in parallel.

Additional info:

Comment 1 Peter Ruan 2013-08-09 23:05:30 UTC
Created attachment 785032 [details]
mcollective.log

Comment 2 Peter Ruan 2013-08-09 23:07:40 UTC
Created attachment 785034 [details]
development.log


Note You need to log in before you can comment on or make changes to this bug.