Bug 995637 - mcollective performance problem when the number of parallel requests = 20.
mcollective performance problem when the number of parallel requests = 20.
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
Unspecified Unspecified
unspecified Severity low
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
Depends On:
  Show dependency treegraph
Reported: 2013-08-09 19:01 EDT by Peter Ruan
Modified: 2015-05-14 20:19 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-02-07 19:46:23 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
mcollective.log (1.52 MB, text/x-log)
2013-08-09 19:05 EDT, Peter Ruan
no flags Details
development.log (3.35 MB, text/x-log)
2013-08-09 19:07 EDT, Peter Ruan
no flags Details

  None (edit)
Description Peter Ruan 2013-08-09 19:01:21 EDT
Description of problem:
  I'm doing parallel cucumber tests against a single user/domain.  I started with 10 threads running 10 different independent scenarios (creating 10 different type of apps and check if I can control it (start/stop/reload, and etc).  This runs fine.  However when I up the threads/test counts to 20, I would get between 3-7 failures out of the 20 apps, with mostly error message like below...  This was run against the latest devenv image.  When I run the same set of tests in PROD, it beaks similarly on 10 threads only. 

[15:25:15] INFO> Shell Command: rhc app create rxulwc jbossas-7 -l pruan@redhat.com -p 'vostok08' --insecure --server ec2-54-211-9-30.compute-1.amazonaws.com
      Application Options
        Namespace:  pruan08
        Cartridges: jbossas-7
        Gear Size:  default
        Scaling:    no
      Creating application 'rxulwc' ... An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift server 'https://ec2-54-211-9-30.compute-1.amazonaws.com/broker/rest/domains/pruan08/applications'.
      [15:30:23] INFO> Exit Status: 1
      [15:30:23] WARN> Expected response not found: Your application 'rxulwc' is now available

[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Got Node Unavailable Exception (pid:32355)
^[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Reference ID: 84263f4f4b6e86f1ba84e059b14901c4 - No nodes available.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. create  10/20 apps of different types in parallel against a devenv.

Actual results:
Failing Scenarios:
cucumber features/demo19.feature:2 # Scenario: Demo19
cucumber features/demo18.feature:2 # Scenario: Demo18
cucumber features/demo6.feature:2 # Scenario: Demo6
cucumber features/demo7.feature:2 # Scenario: Demo7
cucumber features/demo8.feature:2 # Scenario: Demo8
cucumber features/demo12.feature:2 # Scenario: Demo12
cucumber features/demo9.feature:2 # Scenario: Demo9

20 scenarios (7 failed, 13 passed)
500 steps (7 failed, 161 skipped, 332 passed)

Took 585.512743205 seconds

Expected results:
be able to create & control independent apps in parallel.

Additional info:
Comment 1 Peter Ruan 2013-08-09 19:05:30 EDT
Created attachment 785032 [details]
Comment 2 Peter Ruan 2013-08-09 19:07:40 EDT
Created attachment 785034 [details]

Note You need to log in before you can comment on or make changes to this bug.