Description of problem: If trying to create several (for eg: 3) ruby (ruby-1.9 or ruby-1.8) apps at the same time, some of them will fail with error: Creating application 'myrubyxidbz' ... mkdir: cannot create directory `/var/lib/openshift/51e4bc1fdbd93cdbf8000087//.gem': File exists Version-Release number of selected component (if applicable): STG(devenv-stage_406) rhc-1.11.4 How reproducible: always Steps to Reproduce: 1. create 3 ruby apps at the same time in 3 subprocess 2. 3. Actual results: one of the apps' creation will fail with the errors below: mkdir: cannot create directory `/var/lib/openshift/51e4dd4b2587c8c2d2000181//.gem': File exists Expected results: Ruby apps should be created successfully. Additional info:
Concurrent app creates with the same gear UUID could result in this error. Reassigning to the broker team for race condition investigation as the node uses UUIDs supplied by the broker. Please reassign to the node component if new evidence suggests some lower level bug.
QE, Can you please provide the script or precise steps used to reproduce this scenario? Thanks.
Also, assuming you are trying to follow these steps in a devenv, can you also attach the broker and node logs?
Tried creating 8 ruby 1.8 applications in parallel multiple times and got no failures. Also, tried with ruby 1.9 and all attempts were successful. Tried to create applications both using CLI and REST APIs.
(In reply to Dan Mace from comment #2) (In reply to Abhishek Gupta from comment #3) > Also, assuming you are trying to follow these steps in a devenv, can you > also attach the broker and node logs? This was happening on STG(devenv-stage_406), guess you could get log from ops.
(In reply to Dan Mace from comment #2) > QE, > > Can you please provide the script or precise steps used to reproduce this > scenario? > > Thanks. Just found several ruby creation failures while testing against STG( which was matching devenv-stage_406) yesterday, and QE tried to reproduce it, hard to find a consistent way to reproduce it as well, and found if creating several apps at the same time against STG will increase the chance to reproduce it. QE will try if it could still be reproduced today after STG(devenv-stage_410) is deployed.
Tried it on STG(devenv-stage_410) just now, it could still be reproduced by just using 3 terminals to create 3 different ruby apps at the same time(almost since has some latency manually), 1 of 3 is created successfully, the other 2 failed, see detail result below: terminal1#rhc app create app11 ruby-1.8 Application Options ------------------- Namespace: qcong Cartridges: ruby-1.8 Gear Size: default Scaling: no Creating application 'app11' ... done Waiting for your DNS name to be available ... done Initialized empty Git repository in /root/test/app11/.git/ Your application 'app11' is now available. URL: http://app11-qcong.stg.rhcloud.com/ SSH to: 51e5f932dbd93ce4b50001c7.rhcloud.com Git remote: ssh://51e5f932dbd93ce4b50001c7.rhcloud.com/~/git/app11.git/ Cloned to: /root/test/app11 Run 'rhc show-app app11' for more details about your app. terminal2# rhc app create app12 ruby-1.8 Application Options ------------------- Namespace: qcong Cartridges: ruby-1.8 Gear Size: default Scaling: no Creating application 'app12' ... mkdir: cannot create directory `/var/lib/openshift/51e5f9322587c8f1220001e5//.gem': File exists terminal3# rhc app create app13 ruby-1.8 Application Options ------------------- Namespace: qcong Cartridges: ruby-1.8 Gear Size: default Scaling: no Creating application 'app13' ... mkdir: cannot create directory `/var/lib/openshift/51e5f934dbd93c3c630002b0//.gem': File exists
Version-Release number of selected component (if applicable): devenv-stage_406 Steps to reproduce: 1.create 3 ruby apps in 3 terminal almost at the same time Actual result: all the 3 apps are created successfully Expected result: all the 3 apps are created successfully The bug can't be reproduced on a single node.
Version-Release number of selected component (if applicable): devenv-stage_410 Steps to reproduce: 1.create 3 ruby apps in 3 terminal almost at the same time Actual result: all the 3 apps are created successfully Expected result: all the 3 apps are created successfully The bug can't be reproduced on a single node.
The bug was reproducible in today's auto test against STG(devenv-stage_410), and I listed some of the the problematic creations with the corresponding accounts and the error messages in this comment: 1) Account: xtian+test2 Error Msg: Creating application 'rubyre8zz0y' ... mkdir: cannot create directory `/var/lib/openshift/51e5fe9edbd93c3c630002e9//.gem': File exists 2) Account: jizhao+2 Error Msg: Creating application 'rubyzmgh43i' ... mkdir: cannot create directory `/var/lib/openshift/51e60b4f2587c8140a000093//.gem': File exists 3) Account: xtian+test5 Error Msg: Creating application 'p69zkb1rdj' ... mkdir: cannot create directory `/var/lib/openshift/51e626fbdbd93c18480003ed//.gem': File exists 4) Account: xtian+test3 Error Msg: Creating application 'snapshot0ce5' ... mkdir: cannot create directory `/var/lib/openshift/51e610c7dbd93cbc640000f3//.gem': File exists 5) Account: xtian+test5 Error Msg: Creating application 'tailoptionscm0y' ... mkdir: cannot create directory `/var/lib/openshift/51e625d2dbd93cd51d0000ac//.gem': File exists Hope the info helps. Thanks, Zhe Wang
This was occurring due to all cartridges a certain stage node being out of date as a side effect of an unrelated RPM error preventing the cartridge installation from succeeding. The node has been manually repaired. A new bug has been filed to prevent similar situations in the future: https://bugzilla.redhat.com/show_bug.cgi?id=985514
Blocked by bug 985756.
Since the bug 985756 marked as not a bug. And the issue for this bug has been fixed. Ruby package on the broken node has been installed completely. [ruby4-bmengs.stg.rhcloud.com 51e8ae5a2587c8145d000274]\> hostname ex-std-node2.stg.rhcloud.com Move bug to verified.