Description of problem: I'm doing parallel cucumber tests against a single user/domain. I started with 10 threads running 10 different independent scenarios (creating 10 different type of apps and check if I can control it (start/stop/reload, and etc). This runs fine. However when I up the threads/test counts to 20, I would get between 3-7 failures out of the 20 apps, with mostly error message like below... This was run against the latest devenv image. When I run the same set of tests in PROD, it beaks similarly on 10 threads only. [15:25:15] INFO> Shell Command: rhc app create rxulwc jbossas-7 -l pruan -p 'vostok08' --insecure --server ec2-54-211-9-30.compute-1.amazonaws.com Application Options ------------------- Namespace: pruan08 Cartridges: jbossas-7 Gear Size: default Scaling: no Creating application 'rxulwc' ... An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift server 'https://ec2-54-211-9-30.compute-1.amazonaws.com/broker/rest/domains/pruan08/applications'. [15:30:23] INFO> Exit Status: 1 [15:30:23] WARN> Expected response not found: Your application 'rxulwc' is now available [[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Got Node Unavailable Exception (pid:32355) ^[[0;37m2013-08-09 18:28:12.120^[[0m [^[[31mERROR^[[0m] Reference ID: 84263f4f4b6e86f1ba84e059b14901c4 - No nodes available. Version-Release number of selected component (if applicable): devenv How reproducible: always Steps to Reproduce: 1. create 10/20 apps of different types in parallel against a devenv. 2. 3. Actual results: Failing Scenarios: cucumber features/demo19.feature:2 # Scenario: Demo19 cucumber features/demo18.feature:2 # Scenario: Demo18 cucumber features/demo6.feature:2 # Scenario: Demo6 cucumber features/demo7.feature:2 # Scenario: Demo7 cucumber features/demo8.feature:2 # Scenario: Demo8 cucumber features/demo12.feature:2 # Scenario: Demo12 cucumber features/demo9.feature:2 # Scenario: Demo9 20 scenarios (7 failed, 13 passed) 500 steps (7 failed, 161 skipped, 332 passed) Took 585.512743205 seconds Expected results: be able to create & control independent apps in parallel. Additional info:
Created attachment 785032 [details] mcollective.log
Created attachment 785034 [details] development.log