This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 860236 - [fork] restart activemq followed by configure libra-datastore will cause app creation failure
[fork] restart activemq followed by configure libra-datastore will cause app ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Mark Lamourine
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-25 07:10 EDT by Rony Gong
Modified: 2015-05-14 22:05 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-06 13:48:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
app create fail log (22.43 KB, text/plain)
2012-09-25 07:10 EDT, Rony Gong
no flags Details

  None (edit)
Description Rony Gong 2012-09-25 07:10:31 EDT
Created attachment 616973 [details]
app create fail log

Description of problem:
After do "service libra-datastore configure" in instance, show failed when stoping mongdb
Stopping mongod:                                           [FAILED]
but then check the mongd service status is runnning:
[root@ip-10-123-3-119 ~]# service mongod status
mongod (pid 1791) is running...

Version-Release number of selected component (if applicable):
fork_ami_devenv-activemq_143

How reproducible:
always

Steps to Reproduce:

1.ssh into instance by root
2.[root@ip-10-123-3-119 ~]# service libra-datastore configure
Starting mongod:                                           [  OK  ]
MongoDB shell version: 2.0.7
connecting to: test
switched to db openshift_broker_dev
{
    "updatedExisting" : true,
    "n" : 1,
    "lastOp" : NumberLong("5792064170920771585"),
    "connectionId" : 181,
    "err" : null,
    "ok" : 1
}
{
    "_id" : ObjectId("506058fa5c66045a551c869d"),
    "user" : "libra",
    "readOnly" : false,
    "pwd" : "09a543e34e8b9161c89bcd3ba1f82140"
}
bye
Stopping mongod:                                           [FAILED]
3.[root@ip-10-123-3-119 ~]# service activemq restart
Stopping ActiveMQ Broker...
Stopped ActiveMQ Broker.
Starting ActiveMQ Broker...
4. Then create app by CLI
 rhc app create -a perl -t perl-5.10
  

Actual results:
Submitting form:
debug: true
rhlogin: qgong@redhat.com
Contacting https://ec2-50-17-78-245.compute-1.amazonaws.com
Creating application: qperl2 in qgong8
Contacting https://ec2-50-17-78-245.compute-1.amazonaws.com
Problem reported from server. Response code was 500.
DEBUG:
Exit Code: 1
api_c: placeholder
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
API version:    1.1.3
RESULT:
Input uid should be a number

Expected results:
service libra-datastore configure  should not show failed
and could create app success after restart activemq

Additional info:
if go on do "service libra-datastore restart", then could create app success
If only just do  libra-datastore configure, could create app success.
Comment 1 Mark Lamourine 2012-09-25 11:53:30 EDT
Initial investigation indicates a possible race condition.

The activemq service restart returns and indicates that the startup is complete, but in fact it is not yet ready to begin passing messages.

  # service activemq restart
  Stopping ActiveMQ Broker...
  Stopped ActiveMQ Broker.
  Starting ActiveMQ Broker...
  # mco ping

  ---- ping statistics ----
  No responses received
  #

A few moments later 'mco ping' replies with the expected single node.

Attempting an 'rhc app create' immediately after 'service activemq restart' yeilds "no nodes available'.  A few moments later it will get the 'Input uid should be a number' message.

Waiting an additional ~30 seconds allows the command to complete as expected.  I'll see if I can get the timing down and (better still) find a way to be sure communications are working before submitting the next command.
Comment 2 Mark Lamourine 2012-10-04 13:22:36 EDT
I can't find a way (short of attempt) to confirm that the activemq service is responding after restart.

Would it be possible to modify the test logic to add these steps?

* restart
* sleep 60
* attempt
-- on fail
   sleep 30
   retry
   on fail -> fail the test
* continue

- Mark
Comment 3 Rony Gong 2012-10-08 06:38:48 EDT
Retest on devenv_2297
1.ssh into instance
[root@ip-10-93-74-146 ~]# service rhc-datastore configure
Starting mongod:                                           [  OK  ]
MongoDB shell version: 2.0.7
connecting to: test
switched to db openshift_broker_dev
{
	"updatedExisting" : true,
	"n" : 1,
	"lastOp" : NumberLong("5796883871551193089"),
	"connectionId" : 28,
	"err" : null,
	"ok" : 1
}
{
	"_id" : ObjectId("507284f157d910392dad5ec6"),
	"user" : "libra",
	"readOnly" : false,
	"pwd" : "09a543e34e8b9161c89bcd3ba1f82140"
}
bye
Stopping mongod:                                           [FAILED]
2.[root@ip-10-93-74-146 ~]# service activemq restart
Stopping ActiveMQ Broker...
Stopped ActiveMQ Broker.
Starting ActiveMQ Broker...
3.Wait 60s, then create app in CLI
4.[qgong@localhost dev]$ rhc app create q2perl perl-5.10
Password: 

nil
**********
Creating application 'q2perl'
=============================

  Gear Size: default
  Cartridge: perl-5.10
  Namespace: qgong16
  Scaling:   no

Your application's domain name is being propagated worldwide (this might take a minute)...
Initialized empty Git repository in /home/qgong/mytest/dev/q2perl/.git/
done

q2perl @ http://q2perl-qgong16.dev.rhcloud.com/
===============================================
Created: 6:37 AM
   UUID: 0534d2b543794a0ea460062db12b3cce
Git URL: ssh://0534d2b543794a0ea460062db12b3cce@q2perl-qgong16.dev.rhcloud.com/~/git/q2perl.git/
SSH URL: ssh://0534d2b543794a0ea460062db12b3cce@q2perl-qgong16.dev.rhcloud.com

Cartridges:
  perl-5.10
RESULT:
Application q2perl was created.

Note You need to log in before you can comment on or make changes to this bug.