Bug 984818 - ruby-1.8 or 1.9 app creation will be failed if creating several apps in parallel
ruby-1.8 or 1.9 app creation will be failed if creating several apps in par...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Linux
medium Severity medium
: ---
: ---
Assigned To: Dan Mace
libra bugs
:
Depends On: 985756
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-16 02:21 EDT by Lei Zhang
Modified: 2015-05-14 19:24 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-22 11:24:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Lei Zhang 2013-07-16 02:21:59 EDT
Description of problem:
If trying to create several (for eg: 3)  ruby (ruby-1.9 or ruby-1.8) apps at the same time, some of them will fail  with error:
Creating application 'myrubyxidbz' ... mkdir: cannot create directory `/var/lib/openshift/51e4bc1fdbd93cdbf8000087//.gem': File exists

Version-Release number of selected component (if applicable):
STG(devenv-stage_406)
rhc-1.11.4

How reproducible:
always

Steps to Reproduce:
1. create 3 ruby apps at the same time in 3 subprocess
2.
3.

Actual results:
one of the apps' creation will fail with the errors below:
mkdir: cannot create directory `/var/lib/openshift/51e4dd4b2587c8c2d2000181//.gem': File exists

Expected results:
Ruby apps should be created successfully.

Additional info:
Comment 1 Dan Mace 2013-07-16 11:35:01 EDT
Concurrent app creates with the same gear UUID could result in this error. Reassigning to the broker team for race condition investigation as the node uses UUIDs supplied by the broker. Please reassign to the node component if new evidence suggests some lower level bug.
Comment 2 Dan Mace 2013-07-16 14:11:47 EDT
QE,

Can you please provide the script or precise steps used to reproduce this scenario?

Thanks.
Comment 3 Abhishek Gupta 2013-07-16 18:00:12 EDT
Also, assuming you are trying to follow these steps in a devenv, can you also attach the broker and node logs?
Comment 4 Abhishek Gupta 2013-07-16 18:12:18 EDT
Tried creating 8 ruby 1.8 applications in parallel multiple times and got no failures. Also, tried with ruby 1.9 and all attempts were successful. Tried to create applications both using CLI and REST APIs.
Comment 5 Xiaoli Tian 2013-07-16 21:25:43 EDT
(In reply to Dan Mace from comment #2)

(In reply to Abhishek Gupta from comment #3)
> Also, assuming you are trying to follow these steps in a devenv, can you
> also attach the broker and node logs?

This was happening on STG(devenv-stage_406),  guess you could get log from ops.
Comment 6 Xiaoli Tian 2013-07-16 21:33:50 EDT
(In reply to Dan Mace from comment #2)
> QE,
> 
> Can you please provide the script or precise steps used to reproduce this
> scenario?
> 
> Thanks.

Just found several ruby creation failures while testing against STG( which was matching devenv-stage_406) yesterday, and QE tried to reproduce it, hard to find a consistent way to reproduce it as well, and found if creating several apps at the same time against STG will increase the chance to reproduce it.

QE will try if it could still be reproduced today after STG(devenv-stage_410) is deployed.
Comment 7 congqiyuan 2013-07-16 22:00:22 EDT
Tried it on STG(devenv-stage_410) just now, it could still be reproduced by just using 3 terminals to create 3 different ruby apps at the same time(almost since has some latency manually), 1 of 3 is created successfully, the other 2 failed, see detail result below:

terminal1#rhc app create app11 ruby-1.8
Application Options
-------------------
  Namespace:  qcong
  Cartridges: ruby-1.8
  Gear Size:  default
  Scaling:    no

Creating application 'app11' ... done


Waiting for your DNS name to be available ... done

Initialized empty Git repository in /root/test/app11/.git/

Your application 'app11' is now available.

  URL:        http://app11-qcong.stg.rhcloud.com/
  SSH to:     51e5f932dbd93ce4b50001c7@app11-qcong.stg.rhcloud.com
  Git remote: ssh://51e5f932dbd93ce4b50001c7@app11-qcong.stg.rhcloud.com/~/git/app11.git/
  Cloned to:  /root/test/app11

Run 'rhc show-app app11' for more details about your app.


terminal2# rhc app create app12 ruby-1.8
Application Options
-------------------
  Namespace:  qcong
  Cartridges: ruby-1.8
  Gear Size:  default
  Scaling:    no

Creating application 'app12' ... mkdir: cannot create directory `/var/lib/openshift/51e5f9322587c8f1220001e5//.gem': File exists

terminal3# rhc app create app13 ruby-1.8
Application Options
-------------------
  Namespace:  qcong
  Cartridges: ruby-1.8
  Gear Size:  default
  Scaling:    no

Creating application 'app13' ... mkdir: cannot create directory `/var/lib/openshift/51e5f934dbd93c3c630002b0//.gem': File exists
Comment 8 congqiyuan 2013-07-16 22:32:10 EDT
Version-Release number of selected component (if applicable):
devenv-stage_406

Steps to reproduce:
1.create 3 ruby apps in 3 terminal almost at the same time

Actual result:
all the 3 apps are created successfully

Expected result:
all the 3 apps are created successfully

The bug can't be reproduced on a single node.
Comment 9 congqiyuan 2013-07-16 22:33:45 EDT
Version-Release number of selected component (if applicable):
devenv-stage_410

Steps to reproduce:
1.create 3 ruby apps in 3 terminal almost at the same time

Actual result:
all the 3 apps are created successfully

Expected result:
all the 3 apps are created successfully

The bug can't be reproduced on a single node.
Comment 10 Zhe Wang 2013-07-17 02:06:14 EDT
The bug was reproducible in today's auto test against STG(devenv-stage_410), and I listed some of the the problematic creations with the corresponding accounts and the error messages in this comment:

1) Account: xtian+test2@redhat.com
   Error Msg: Creating application 'rubyre8zz0y' ... mkdir: cannot create directory `/var/lib/openshift/51e5fe9edbd93c3c630002e9//.gem': File exists

2) Account: jizhao+2@redhat.com
   Error Msg: Creating application 'rubyzmgh43i' ... mkdir: cannot create directory `/var/lib/openshift/51e60b4f2587c8140a000093//.gem': File exists

3) Account: xtian+test5@redhat.com
   Error Msg: Creating application 'p69zkb1rdj' ... mkdir: cannot create directory `/var/lib/openshift/51e626fbdbd93c18480003ed//.gem': File exists

4) Account: xtian+test3@redhat.com
   Error Msg: Creating application 'snapshot0ce5' ... mkdir: cannot create directory `/var/lib/openshift/51e610c7dbd93cbc640000f3//.gem': File exists

5) Account: xtian+test5@redhat.com
   Error Msg: Creating application 'tailoptionscm0y' ... mkdir: cannot create directory `/var/lib/openshift/51e625d2dbd93cd51d0000ac//.gem': File exists

Hope the info helps.

Thanks,
Zhe Wang
Comment 11 Dan Mace 2013-07-17 12:23:23 EDT
This was occurring due to all cartridges a certain stage node being out of date as a side effect of an unrelated RPM error preventing the cartridge installation from succeeding. The node has been manually repaired.

A new bug has been filed to prevent similar situations in the future: https://bugzilla.redhat.com/show_bug.cgi?id=985514
Comment 12 Meng Bo 2013-07-18 04:36:03 EDT
Blocked by bug 985756.
Comment 13 Meng Bo 2013-07-18 23:15:37 EDT
Since the bug 985756 marked as not a bug.

And the issue for this bug has been fixed.

Ruby package on the broken node has been installed completely.

[ruby4-bmengs.stg.rhcloud.com 51e8ae5a2587c8145d000274]\> hostname
ex-std-node2.stg.rhcloud.com


Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.