Created attachment 616973 [details] app create fail log Description of problem: After do "service libra-datastore configure" in instance, show failed when stoping mongdb Stopping mongod: [FAILED] but then check the mongd service status is runnning: [root@ip-10-123-3-119 ~]# service mongod status mongod (pid 1791) is running... Version-Release number of selected component (if applicable): fork_ami_devenv-activemq_143 How reproducible: always Steps to Reproduce: 1.ssh into instance by root 2.[root@ip-10-123-3-119 ~]# service libra-datastore configure Starting mongod: [ OK ] MongoDB shell version: 2.0.7 connecting to: test switched to db openshift_broker_dev { "updatedExisting" : true, "n" : 1, "lastOp" : NumberLong("5792064170920771585"), "connectionId" : 181, "err" : null, "ok" : 1 } { "_id" : ObjectId("506058fa5c66045a551c869d"), "user" : "libra", "readOnly" : false, "pwd" : "09a543e34e8b9161c89bcd3ba1f82140" } bye Stopping mongod: [FAILED] 3.[root@ip-10-123-3-119 ~]# service activemq restart Stopping ActiveMQ Broker... Stopped ActiveMQ Broker. Starting ActiveMQ Broker... 4. Then create app by CLI rhc app create -a perl -t perl-5.10 Actual results: Submitting form: debug: true rhlogin: qgong Contacting https://ec2-50-17-78-245.compute-1.amazonaws.com Creating application: qperl2 in qgong8 Contacting https://ec2-50-17-78-245.compute-1.amazonaws.com Problem reported from server. Response code was 500. DEBUG: Exit Code: 1 api_c: placeholder broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi API version: 1.1.3 RESULT: Input uid should be a number Expected results: service libra-datastore configure should not show failed and could create app success after restart activemq Additional info: if go on do "service libra-datastore restart", then could create app success If only just do libra-datastore configure, could create app success.
Initial investigation indicates a possible race condition. The activemq service restart returns and indicates that the startup is complete, but in fact it is not yet ready to begin passing messages. # service activemq restart Stopping ActiveMQ Broker... Stopped ActiveMQ Broker. Starting ActiveMQ Broker... # mco ping ---- ping statistics ---- No responses received # A few moments later 'mco ping' replies with the expected single node. Attempting an 'rhc app create' immediately after 'service activemq restart' yeilds "no nodes available'. A few moments later it will get the 'Input uid should be a number' message. Waiting an additional ~30 seconds allows the command to complete as expected. I'll see if I can get the timing down and (better still) find a way to be sure communications are working before submitting the next command.
I can't find a way (short of attempt) to confirm that the activemq service is responding after restart. Would it be possible to modify the test logic to add these steps? * restart * sleep 60 * attempt -- on fail sleep 30 retry on fail -> fail the test * continue - Mark
Retest on devenv_2297 1.ssh into instance [root@ip-10-93-74-146 ~]# service rhc-datastore configure Starting mongod: [ OK ] MongoDB shell version: 2.0.7 connecting to: test switched to db openshift_broker_dev { "updatedExisting" : true, "n" : 1, "lastOp" : NumberLong("5796883871551193089"), "connectionId" : 28, "err" : null, "ok" : 1 } { "_id" : ObjectId("507284f157d910392dad5ec6"), "user" : "libra", "readOnly" : false, "pwd" : "09a543e34e8b9161c89bcd3ba1f82140" } bye Stopping mongod: [FAILED] 2.[root@ip-10-93-74-146 ~]# service activemq restart Stopping ActiveMQ Broker... Stopped ActiveMQ Broker. Starting ActiveMQ Broker... 3.Wait 60s, then create app in CLI 4.[qgong@localhost dev]$ rhc app create q2perl perl-5.10 Password: nil ********** Creating application 'q2perl' ============================= Gear Size: default Cartridge: perl-5.10 Namespace: qgong16 Scaling: no Your application's domain name is being propagated worldwide (this might take a minute)... Initialized empty Git repository in /home/qgong/mytest/dev/q2perl/.git/ done q2perl @ http://q2perl-qgong16.dev.rhcloud.com/ =============================================== Created: 6:37 AM UUID: 0534d2b543794a0ea460062db12b3cce Git URL: ssh://0534d2b543794a0ea460062db12b3cce.rhcloud.com/~/git/q2perl.git/ SSH URL: ssh://0534d2b543794a0ea460062db12b3cce.rhcloud.com Cartridges: perl-5.10 RESULT: Application q2perl was created.