Bug 1100411

Summary:	ceilometer attempts to start before mongo service is ready
Product:	Red Hat OpenStack	Reporter:	Jason Guiditta <jguiditt>
Component:	openstack-foreman-installer	Assignee:	Jason Guiditta <jguiditt>
Status:	CLOSED ERRATA	QA Contact:	Omri Hochman <ohochman>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.0	CC:	acathrow, ajeain, apevec, breeler, e.vnn1, fdinitto, fpercoco, jguiditt, lhh, mburns, mmagr, morazi, ohochman, pbrady, rhos-maint, sclewis, srevivo, yeylon
Target Milestone:	z4	Keywords:	ZStream
Target Release:	4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-foreman-installer-1.0.12-1.el6ost	Doc Type:	Bug Fix
Doc Text:	Cause: Ceilometer would attempt to create its database before the mongo service was up and running. Consequence: Puppet run would fail. Fix: Changed puppet dependency to wait for mongo service to return success. Result: No error on puppet run related to mongo service not yet being started.	Story Points:	---
Clone Of:	1066408	Environment:
Last Closed:	2014-05-29 20:33:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1063659, 1066408
Bug Blocks:	1032769, 1040649

Description Jason Guiditta 2014-05-22 18:52:31 UTC

+++ This bug was initially created as a clone of Bug #1066408 +++

--- Additional comment from Fabio Massimo Di Nitto on 2014-02-11 03:18:40 EST ---

Please be aware that this applies also to mongodb init script.

This causes Ceilometer to fail on initial startup, when mongodb initializes its database:
/var/log/ceilometer/ceilometer-dbsync.log
------------------------------------------------
INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)]
CRITICAL ceilometer [-] could not connect to localhost:27017: [Errno 111] ECONNREFUSED
TRACE ceilometer Traceback (most recent call last):
TRACE ceilometer   File "/usr/bin/ceilometer-dbsync", line 10, in <module>
TRACE ceilometer     sys.exit(dbsync())
TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 139, in dbsync
TRACE ceilometer     get_connection(cfg.CONF).upgrade()
TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 81, in get_connection
TRACE ceilometer     return get_engine(conf).get_connection(conf)
TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 77, in get_connection
TRACE ceilometer     return Connection(conf)
TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 346, in __init__
TRACE ceilometer     self.conn = self.CONNECTION_POOL.connect(url)
TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 163, in connect
TRACE ceilometer     safe=True)
TRACE ceilometer   File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", line 337, in __init__
TRACE ceilometer     raise ConnectionFailure(str(e))
TRACE ceilometer ConnectionFailure: could not connect to localhost:27017: [Errno 111] ECONNREFUSED
TRACE ceilometer

--- Additional comment from RHEL Product and Program Management on 2014-02-18 05:46:43 EST ---

Since this issue was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Alan Pevec on 2014-02-18 06:03:08 EST ---

<apevec> fpercoco, ideally, mongodb would notify via system notification when it's ready
<apevec> http://www.freedesktop.org/software/systemd/man/sd_notify.html
<fpercoco> apevec: yeah, I think I saw some threads w.r.t making mongodb play nice with systemd

NB: current version in Fedora is Type=Forking
  http://pkgs.fedoraproject.org/cgit/mongodb.git/tree/mongod.service is 

<fpercoco> apevec: I'd say looping until socket responds is enough. Another option would be to run something like `mongo --eval "db.version()"`

<eglynn> apevec: how does a simple "mongo --port 27017" react while mongodb is still being fired up?
<eglynn> apevec: ... i.e. fail-fast or busy hang awaiting a connection
<fpercoco> eglynn: it should fail
<fpercoco> not connections are accepted until the preallocation of journal files ends

--- Additional comment from Alan Pevec on 2014-02-18 06:10:33 EST ---

FWIW Fedora systemd mongodb.service[1] with Type=Forking has the same issue[2] - see last section "Update for Fedora 18"

[1] http://pkgs.fedoraproject.org/cgit/mongodb.git/tree/mongod.service
[2] http://stackoverflow.com/questions/11081822/mongodb-service-not-running-in-fedora

--- Additional comment from Omri Hochman on 2014-02-18 06:13:19 EST ---

Ceilometer mongodb error occurred to me during foreman deployed controller with nova network (Havana-A2 from 2014-02-17.1).

puppet agent output :
----------------------
Notice: /Stage[main]/Rsync::Server/Exec[compile fragments]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]/ensure: created
Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Service[ceilometer-collector]
Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Service[ceilometer-agent-central]
Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Exec[ceilometer-dbsync]
Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Exec[ceilometer-dbsync]
Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Service[ceilometer-api]
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.313 17820 INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)]
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 CRITICAL ceilometer [-] could not connect to localhost:27017: [Errno 111] ECONNREFUSED
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer Traceback (most recent call last):
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/bin/ceilometer-dbsync", line 10, in <module>
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     sys.exit(dbsync())
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 139, in dbsync
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     get_connection(cfg.CONF).upgrade()
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 81, in get_connection
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     return get_engine(conf).get_connection(conf)
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 77, in get_connection
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     return Connection(conf)
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 346, in __init__
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     self.conn = self.CONNECTION_POOL.connect(url)
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 163, in connect
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     safe=True)
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer   File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", line 337, in __init__
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer     raise ConnectionFailure(str(e))
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer ConnectionFailure: could not connect to localhost:27017: [Errno 111] ECONNREFUSED
Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer 
Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]
Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]



As well same error at :/var/log/ceilometer/ceilometer-dbsync.log

--- Additional comment from Alan Pevec on 2014-02-27 15:34:18 EST ---

Workaround in puppet-ceilometer could be to loop until mongo --eval "db.version()" succeeds before running ceilometer db sync.

--- Additional comment from Fabio Massimo Di Nitto on 2014-02-27 15:56:46 EST ---

(In reply to Alan Pevec from comment #5)
> Workaround in puppet-ceilometer could be to loop until mongo --eval
> "db.version()" succeeds before running ceilometer db sync.

There is no such workaround for HA or ceilometer db sync.

The script needs to return the correct status.

--- Additional comment from Lon Hohberger on 2014-03-11 13:05:21 EDT ---



--- Additional comment from Lon Hohberger on 2014-03-11 13:15:59 EDT ---



--- Additional comment from Pádraig Brady on 2014-04-09 09:22:58 EDT ---

I presume these are for merging back to Fedora/EPEL too?

--- Additional comment from Flavio Percoco on 2014-04-14 12:10:38 EDT ---

@pbrady Yeah, it should probably merged back upstream too.

--- Additional comment from Flavio Percoco on 2014-04-14 12:16:46 EDT ---



--- Additional comment from Lon Hohberger on 2014-04-14 16:56:11 EDT ---

It sounds like my patch broke a case where an IP wasn't set in the config file.

This will have to be conditionalized.

--- Additional comment from errata-xmlrpc on 2014-04-14 17:29:24 EDT ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHSA-2014:17483-01
https://errata.devel.redhat.com/advisory/17483

--- Additional comment from errata-xmlrpc on 2014-04-14 17:30:27 EDT ---

This bug has been dropped from advisory RHSA-2014:17483 by Lon Hohberger (lhh)

--- Additional comment from errata-xmlrpc on 2014-04-14 18:10:37 EDT ---

Bug report changed from MODIFIED to ON_QA status by the Errata System: 
Advisory RHBA-2014:17449-01: 
Changed by: Lon Hohberger (lhh)
http://errata.devel.redhat.com/advisory/17449

--- Additional comment from Lon Hohberger on 2014-04-15 14:08:14 EDT ---

My initscript assumes presence of mongodb in order to operate (using the mongostat command).

Thus, we can either make the server package require 'mongodb' or conditionalize the initscript on the presence of the mongostat command.

--- Additional comment from Lon Hohberger on 2014-04-15 14:22:10 EDT ---

I don't think making the client package optional would allow the bug to be fixed.

For now, I'll rebuild and mongodb-server will require mongodb (but not the other way around).

--- Additional comment from Lon Hohberger on 2014-04-15 14:27:54 EDT ---

This didn't come up with 2.2.4 because it already had the fully-versioned requirement on mongodb-server requiring mongodb.

--- Additional comment from Omri Hochman on 2014-04-23 05:46:56 EDT ---

Issue reproduced with mongodb-2.4.6-2.el6ost.x86_64.

When attempted to deploy "controller" (Controller with : neutron or nova-network)  using foreman Havana A4 2014-04-15.5 

While running puppet agent -t -v  on the client machine --> getting the following error and the client remain in Error after deployment:

The Results:
-------------
Foreman-client will remain in status Error 'E' in foreman-server-GUI (*until running the puppet for the second time*) 

/var/log/messages :
--------------------
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]/ensure) created
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Service[ceilometer-collector]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Service[ceilometer-agent-central]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Exec[ceilometer-dbsync]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Exec[ceilometer-dbsync]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Service[ceilometer-api]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.673 4472 INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 CRITICAL ceilometer [-] could not connect to localhost:27017: [Errno 111] ECONNREFUSED
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer Traceback (most recent call last):
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/bin/ceilometer-dbsync", line 10, in <module>
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     sys.exit(dbsync())
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 139, in dbsync
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     get_connection(cfg.CONF).upgrade()
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 81, in get_connection
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     return get_engine(conf).get_connection(conf)
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 78, in get_connection
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     return Connection(conf)
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 351, in __init__
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     self.conn = self.CONNECTION_POOL.connect(url)
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 164, in connect
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     safe=True)
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer   File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", line 337, in __init__
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer     raise ConnectionFailure(str(e))
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer ConnectionFailure: could not connect to localhost:27017: [Errno 111] ECONNREFUSED
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer 
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]) Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]
Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]) ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]
Apr 22 16:45:45 puma01 yum[4480]: Installed: openstack-ceilometer-collector-2013.2.3-1.el6ost.noarch
Apr 22 16:45:46 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Collector/Package[ceilometer-collector]/ensure) created

--- Additional comment from Flavio Percoco on 2014-04-23 06:01:29 EDT ---

@Omri

Are there any logs from mongodb's installation?

--- Additional comment from Omri Hochman on 2014-04-23 06:10:23 EDT ---

Flavio : I've Added mongodb.log (attached)

--- Additional comment from Alan Pevec on 2014-04-23 08:08:01 EDT ---

That looks like missing dependency in ceilo puppet recipe, Ceilometer::Db starts at 16:45 while mongodb is started only at 16:50 ?

--- Additional comment from Lon Hohberger on 2014-04-23 08:20:46 EDT ---

Yes, that looks right, Alan.

The mongodb logs indicate that mongodb has started - this out-of-order problem isn't related to the initscript changes.

--- Additional comment from Flavio Percoco on 2014-04-23 08:52:15 EDT ---

Alan beat me on this. It sounds like a possible cause.

Omri, are these logs taken from the first or second execution? 

I think a stronger dependency on mongodb is required in the manifest. AFAICS, the mongodb manifest is put in the list of manifests before ceilo's but that doesn't seem to be enough.

--- Additional comment from Omri Hochman on 2014-04-23 09:25:00 EDT ---

(In reply to Flavio Percoco from comment #24)

> Omri, are these logs taken from the first or second execution? 

It took from the same run of https://bugzilla.redhat.com/show_bug.cgi?id=1066408#c19

Just in case - I'll attach the full messages.

--- Additional comment from Omri Hochman on 2014-04-23 09:25:46 EDT ---



--- Additional comment from Omri Hochman on 2014-04-24 10:17:56 EDT ---

Steps to reproduce: 
--------------------
1) Install foreman-server using foreman_server.sh
2) Register foreman-client against foreman-server (using foreman_client.sh)
3) Associate the Client with Controller host group (can be either neutron or nova-network) 
4) run puppet agent on the client machine (puppet agent -t -v)

Results:
---------

- Error display during puppet agent running:
" Ceilometer::Db/Exec[ceilometer-dbsync]) Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]:"

- The client will remain in Error - 'E' in the foreman-server GUI.  

Workaround: 
------------
Run 'puppet agent -t -v' - for the second time.

--- Additional comment from Martin Magr on 2014-04-24 10:45:48 EDT ---

Looking at the quickstack manifest [1] and Havana packstack template [2], both have mongodb(::server) requirement on ceilometer::db (where sync-db happens), so it should be ok from Puppet point of view. But unfortunately from Omri's log [3] I found that mongodb is started after ceilometer dbsync. Not sure why Puppet ignores require. Will have to investigate that.

[1] https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/ceilometer_controller.pp
[2] https://github.com/stackforge/packstack/blob/havana/packstack/puppet/templates/ceilometer.pp
[3] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.673 4472 INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)]
Apr 22 16:50:41 puma01 puppet-agent[3736]: (/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure) change from stopped to running failed: Could not start Service[mongodb]: Execution of '/sbin/service mongod start' returned 1:

--- Additional comment from RHEL Product and Program Management on 2014-04-24 11:11:46 EDT ---

This bug report previously had all acks and release flag approved.
However since at least one of its acks has been changed, the
release flag has been reset to ? by the bugbot (pm-rhel).  The
ack needs to become approved before the release flag can become
approved again.

--- Additional comment from Martin Magr on 2014-04-25 08:26:31 EDT ---

I wasn't able to reproduce it, but I modified quickstack manifest to be the same regarding MongoDB/Ceilometer dependency and created a pull request:

https://github.com/redhat-openstack/astapor/pull/191

Please check it out, test it and let Omri some rpm so he can try if it helped.

--- Additional comment from Mike Orazi on 2014-04-30 09:35:46 EDT ---

Moving this to A5.  The upstream patch looks to have some complications that we need to look through before we can pull this into a release.

--- Additional comment from errata-xmlrpc on 2014-05-01 10:00:00 EDT ---

This bug has been dropped from advisory RHBA-2014:17449 by Lon Hohberger (lhh)

Comment 1 Jason Guiditta 2014-05-22 18:57:24 UTC

Patch posted, tested on Omri's machine:

https://github.com/redhat-openstack/astapor/pull/235

Comment 3 Omri Hochman 2014-05-22 23:34:10 UTC

Verified :(with Puddle: 2014-05-22.1 )
mongodb-2.4.6-4.el6ost.x86_64
openstack-foreman-installer-1.0.12-1.el6ost.noarch

Comment 4 Alan Pevec 2014-05-23 10:15:03 UTC

Does the fix
+        require             => Service['mongod'],

need to be applied to Packstack manifest too?

Comment 5 Alan Pevec 2014-05-23 11:49:42 UTC

I've added a comment in https://github.com/jguiditta/astapor/commit/4a086d56593729203836abc98581f21acb806911

> Packstack manifest[1] uses
> require => Class['mongodb::server'],
> Is there difference?

Comment 6 Martin Magr 2014-05-23 12:51:39 UTC

Alan, we had separated MongoDB installation to it's own manifest in Packstack. this manifest is applied before Ceilometer's manifest, which solved the issue in our case.

Comment 9 errata-xmlrpc 2014-05-29 20:33:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0517.html