Bug 844950

Summary:

Sometimes failed to create 2 scalable jbossas app at the same time using different users

Product:

OKD

Reporter:

jizhao

Component:

Containers

Assignee:

Rob Millner <rmillner>

Status:

CLOSED WORKSFORME

QA Contact:

libra bugs <libra-bugs>

Severity:

low

Docs Contact:

Priority:

high

Version:

2.x

CC:

cgole, mfisher, mmcgrath, szhou

Target Milestone:

---

Keywords:

Triaged

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

devenv_1933

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2012-08-29 18:49:49 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
development.log	none
development_new.log	none

Description jizhao 2012-08-01 10:51:32 UTC

Description of problem:
Use 2 different users to create 2 scalable jbossas app at the same time. Sometimes the creation will fail because of "Node execution failure (invalid exit code from node)."

Version-Release number of selected component (if applicable):
devenv_1931

How reproducible:
Sometimes

Steps to Reproduce:
1.Create 2 scalable jbossas app at the same time
$ rhc app create -a app1 -t jbossas-7 -s -l <rhlogin1>
$ rhc app create -a app2 -t jbossas-7 -s -l <rhlogin2>
2.Check development.log on the server
3.
  
Actual results:
I've attached the development.log
Sometimes the creation fails.
<--------------client output--------------->
Contacting https://ec2-23-22-239-132.compute-1.amazonaws.com
Creating application: app2 in jizhao70
Problem reported from server. Response code was 500.

MESSAGES:
Failed to create application app2 due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
<------------------------------------------>

Expected results:
App should be created successfully.

Additional info:

Comment 1 jizhao 2012-08-01 10:52:26 UTC

Created attachment 601709 [details]
development.log

Comment 2 Rob Millner 2012-08-01 20:31:00 UTC

Unable to reproduce the problem after 10 tries; however, the 500 error was generated by an exception raised from inside the code due to the failure of app_ctl.sh

#<NoMethodError: undefined method `cart_commands' for [{:args=>[], :command=>"SYSTEM_SSH_KEY_REMOVE"}]:Array>

/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.14.14/lib/stickshift-controller/app/models/application.rb:1349

Comment 3 Rob Millner 2012-08-01 20:50:24 UTC

Pull request to resolve the exception.

https://github.com/openshift/crankcase/pull/314

Comment 4 Rob Millner 2012-08-01 22:34:02 UTC

Additional bug fixes were required.  The exception should be fixed in devenv_1933.

Comment 5 Chandrika Gole 2012-08-02 00:46:37 UTC

I am still seeing this on devenv1933 and I saw it the first time itself. Attaching development.log


1.Create 2 scalable jbossas app at the same time
$ rhc app create -a app1 -t jbossas-7 -s -l <rhlogin1>
$ rhc app create -a app2 -t jbossas-7 -s -l <rhlogin2>
2.Check development.log on the server

Comment 6 Chandrika Gole 2012-08-02 00:47:00 UTC

Created attachment 601846 [details]
development_new.log

Comment 7 Rob Millner 2012-08-02 03:28:04 UTC

It looks like the haproxy configure hook is failing for some reason:

DEBUG: server results: SSH_KEY_ADD: AAAAB3NzaC1yc2EAAAABIwAAAQEAy1vdhhL/tt2gabawSEvil95SSqxDg2JD94MqlZ
A70Y/eY9hYshmzzNsrIgQzxvA+DKqBGp3ehUwz1RIHuBUb6NHoD2e9lZJGBEWYMU39D10GQv+lFKYUyCHI9OY6w+m3Mc/JmcL2kjX4
zq0g68Y/RtO8Co9eBdLcn0DkDf3egxAjI55G/47bnhTuoak7g2ArkVESLlBrtceN7LMgUDbshJlCgjuk4rspCQ2pnR9fgvBaMIWb4O
ArEGncdUbmNy0txP2JgDEwx/TaHvZFjKFSXHvRJWpL0k60g5iz+Ja+yb00A9DWZhij64tGHZ2gVycHAEIPPBMa9JElazaIAzRg4w==


DEBUG: server results: BROKER_AUTH_KEY_ADD: 

DEBUG: server results: Failed to start /var/lib/stickshift/3f69f4c7ac564c12b5336b0e2d0ed5b4/jbossas-7/jb1_ctl.sh

DEBUG: Cartridge command embedded/haproxy-1.4::configure exitcode = 121


Will keep trying to reproduce.

Comment 8 Chandrika Gole 2012-08-02 19:38:53 UTC

Not seeing this anymore.. 

1.Create 2 scalable jbossas app at the same time
$ rhc app create -a app1 -t jbossas-7 -s -l cgole
$ rhc app create -a app2 -t jbossas-7 -s -l cgole+1
2.Check development.log on the server

Comment 9 Rob Millner 2012-08-02 20:12:58 UTC

Marking low due to the difficulty reproducing.

Comment 10 Mike McGrath 2012-08-29 18:49:49 UTC

We've not heard of any reports of this happening in prod either.  Closing bug for cleanup, please re-open if the issue still exists.