Bug 844950 - Sometimes failed to create 2 scalable jbossas app at the same time using different users
Sometimes failed to create 2 scalable jbossas app at the same time using diff...
Status: CLOSED WORKSFORME
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
high Severity low
: ---
: ---
Assigned To: Rob Millner
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-01 06:51 EDT by jizhao
Modified: 2015-05-14 18:57 EDT (History)
4 users (show)

See Also:
Fixed In Version: devenv_1933
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-29 14:49:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
development.log (79.99 KB, text/x-log)
2012-08-01 06:52 EDT, jizhao
no flags Details
development_new.log (130.31 KB, application/octet-stream)
2012-08-01 20:47 EDT, Chandrika Gole
no flags Details

  None (edit)
Description jizhao 2012-08-01 06:51:32 EDT
Description of problem:
Use 2 different users to create 2 scalable jbossas app at the same time. Sometimes the creation will fail because of "Node execution failure (invalid exit code from node)."

Version-Release number of selected component (if applicable):
devenv_1931

How reproducible:
Sometimes

Steps to Reproduce:
1.Create 2 scalable jbossas app at the same time
$ rhc app create -a app1 -t jbossas-7 -s -l <rhlogin1>
$ rhc app create -a app2 -t jbossas-7 -s -l <rhlogin2>
2.Check development.log on the server
3.
  
Actual results:
I've attached the development.log
Sometimes the creation fails.
<--------------client output--------------->
Contacting https://ec2-23-22-239-132.compute-1.amazonaws.com
Creating application: app2 in jizhao70
Problem reported from server. Response code was 500.

MESSAGES:
Failed to create application app2 due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
<------------------------------------------>

Expected results:
App should be created successfully.

Additional info:
Comment 1 jizhao 2012-08-01 06:52:26 EDT
Created attachment 601709 [details]
development.log
Comment 2 Rob Millner 2012-08-01 16:31:00 EDT
Unable to reproduce the problem after 10 tries; however, the 500 error was generated by an exception raised from inside the code due to the failure of app_ctl.sh

#<NoMethodError: undefined method `cart_commands' for [{:args=>[], :command=>"SYSTEM_SSH_KEY_REMOVE"}]:Array>

/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.14.14/lib/stickshift-controller/app/models/application.rb:1349
Comment 3 Rob Millner 2012-08-01 16:50:24 EDT
Pull request to resolve the exception.

https://github.com/openshift/crankcase/pull/314
Comment 4 Rob Millner 2012-08-01 18:34:02 EDT
Additional bug fixes were required.  The exception should be fixed in devenv_1933.
Comment 5 Chandrika Gole 2012-08-01 20:46:37 EDT
I am still seeing this on devenv1933 and I saw it the first time itself. Attaching development.log


1.Create 2 scalable jbossas app at the same time
$ rhc app create -a app1 -t jbossas-7 -s -l <rhlogin1>
$ rhc app create -a app2 -t jbossas-7 -s -l <rhlogin2>
2.Check development.log on the server
Comment 6 Chandrika Gole 2012-08-01 20:47:00 EDT
Created attachment 601846 [details]
development_new.log
Comment 7 Rob Millner 2012-08-01 23:28:04 EDT
It looks like the haproxy configure hook is failing for some reason:

DEBUG: server results: SSH_KEY_ADD: AAAAB3NzaC1yc2EAAAABIwAAAQEAy1vdhhL/tt2gabawSEvil95SSqxDg2JD94MqlZ
A70Y/eY9hYshmzzNsrIgQzxvA+DKqBGp3ehUwz1RIHuBUb6NHoD2e9lZJGBEWYMU39D10GQv+lFKYUyCHI9OY6w+m3Mc/JmcL2kjX4
zq0g68Y/RtO8Co9eBdLcn0DkDf3egxAjI55G/47bnhTuoak7g2ArkVESLlBrtceN7LMgUDbshJlCgjuk4rspCQ2pnR9fgvBaMIWb4O
ArEGncdUbmNy0txP2JgDEwx/TaHvZFjKFSXHvRJWpL0k60g5iz+Ja+yb00A9DWZhij64tGHZ2gVycHAEIPPBMa9JElazaIAzRg4w==


DEBUG: server results: BROKER_AUTH_KEY_ADD: 

DEBUG: server results: Failed to start /var/lib/stickshift/3f69f4c7ac564c12b5336b0e2d0ed5b4/jbossas-7/jb1_ctl.sh

DEBUG: Cartridge command embedded/haproxy-1.4::configure exitcode = 121


Will keep trying to reproduce.
Comment 8 Chandrika Gole 2012-08-02 15:38:53 EDT
Not seeing this anymore.. 

1.Create 2 scalable jbossas app at the same time
$ rhc app create -a app1 -t jbossas-7 -s -l cgole@redhat.com
$ rhc app create -a app2 -t jbossas-7 -s -l cgole+1@redhat.com
2.Check development.log on the server
Comment 9 Rob Millner 2012-08-02 16:12:58 EDT
Marking low due to the difficulty reproducing.
Comment 10 Mike McGrath 2012-08-29 14:49:49 EDT
We've not heard of any reports of this happening in prod either.  Closing bug for cleanup, please re-open if the issue still exists.

Note You need to log in before you can comment on or make changes to this bug.