Red Hat Bugzilla – Bug 995637
mcollective performance problem when the number of parallel requests = 20.
Last modified: 2015-05-14 20:19:47 EDT
Description of problem:
I'm doing parallel cucumber tests against a single user/domain. I started with 10 threads running 10 different independent scenarios (creating 10 different type of apps and check if I can control it (start/stop/reload, and etc). This runs fine. However when I up the threads/test counts to 20, I would get between 3-7 failures out of the 20 apps, with mostly error message like below... This was run against the latest devenv image. When I run the same set of tests in PROD, it beaks similarly on 10 threads only.
[15:25:15] INFO> Shell Command: rhc app create rxulwc jbossas-7 -l email@example.com -p 'vostok08' --insecure --server ec2-54-211-9-30.compute-1.amazonaws.com
Gear Size: default
Creating application 'rxulwc' ... An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift server 'https://ec2-54-211-9-30.compute-1.amazonaws.com/broker/rest/domains/pruan08/applications'.
[15:30:23] INFO> Exit Status: 1
[15:30:23] WARN> Expected response not found: Your application 'rxulwc' is now available
[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Got Node Unavailable Exception (pid:32355)
^[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Reference ID: 84263f4f4b6e86f1ba84e059b14901c4 - No nodes available.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. create 10/20 apps of different types in parallel against a devenv.
cucumber features/demo19.feature:2 # Scenario: Demo19
cucumber features/demo18.feature:2 # Scenario: Demo18
cucumber features/demo6.feature:2 # Scenario: Demo6
cucumber features/demo7.feature:2 # Scenario: Demo7
cucumber features/demo8.feature:2 # Scenario: Demo8
cucumber features/demo12.feature:2 # Scenario: Demo12
cucumber features/demo9.feature:2 # Scenario: Demo9
20 scenarios (7 failed, 13 passed)
500 steps (7 failed, 161 skipped, 332 passed)
Took 585.512743205 seconds
be able to create & control independent apps in parallel.
Created attachment 785032 [details]
Created attachment 785034 [details]