--- Additional comment from Fabio Massimo Di Nitto on 2014-02-11 03:18:40 EST --- Please be aware that this applies also to mongodb init script. This causes Ceilometer to fail on initial startup, when mongodb initializes its database: /var/log/ceilometer/ceilometer-dbsync.log ------------------------------------------------ INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)] CRITICAL ceilometer [-] could not connect to localhost:27017: [Errno 111] ECONNREFUSED TRACE ceilometer Traceback (most recent call last): TRACE ceilometer File "/usr/bin/ceilometer-dbsync", line 10, in <module> TRACE ceilometer sys.exit(dbsync()) TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 139, in dbsync TRACE ceilometer get_connection(cfg.CONF).upgrade() TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 81, in get_connection TRACE ceilometer return get_engine(conf).get_connection(conf) TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 77, in get_connection TRACE ceilometer return Connection(conf) TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 346, in __init__ TRACE ceilometer self.conn = self.CONNECTION_POOL.connect(url) TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 163, in connect TRACE ceilometer safe=True) TRACE ceilometer File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", line 337, in __init__ TRACE ceilometer raise ConnectionFailure(str(e)) TRACE ceilometer ConnectionFailure: could not connect to localhost:27017: [Errno 111] ECONNREFUSED TRACE ceilometer
FWIW Fedora systemd mongodb.service[1] with Type=Forking has the same issue[2] - see last section "Update for Fedora 18" [1] http://pkgs.fedoraproject.org/cgit/mongodb.git/tree/mongod.service [2] http://stackoverflow.com/questions/11081822/mongodb-service-not-running-in-fedora
Ceilometer mongodb error occurred to me during foreman deployed controller with nova network (Havana-A2 from 2014-02-17.1). puppet agent output : ---------------------- Notice: /Stage[main]/Rsync::Server/Exec[compile fragments]: Triggered 'refresh' from 2 events Notice: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]/ensure: created Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Service[ceilometer-collector] Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Service[ceilometer-agent-central] Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Exec[ceilometer-dbsync] Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Exec[ceilometer-dbsync] Info: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]: Scheduling refresh of Service[ceilometer-api] Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.313 17820 INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)] Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 CRITICAL ceilometer [-] could not connect to localhost:27017: [Errno 111] ECONNREFUSED Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer Traceback (most recent call last): Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/bin/ceilometer-dbsync", line 10, in <module> Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer sys.exit(dbsync()) Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 139, in dbsync Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer get_connection(cfg.CONF).upgrade() Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 81, in get_connection Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer return get_engine(conf).get_connection(conf) Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 77, in get_connection Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer return Connection(conf) Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 346, in __init__ Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer self.conn = self.CONNECTION_POOL.connect(url) Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 163, in connect Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer safe=True) Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", line 337, in __init__ Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer raise ConnectionFailure(str(e)) Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer ConnectionFailure: could not connect to localhost:27017: [Errno 111] ECONNREFUSED Notice: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns: 2014-02-18 12:06:34.326 17820 TRACE ceilometer Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] As well same error at :/var/log/ceilometer/ceilometer-dbsync.log
Workaround in puppet-ceilometer could be to loop until mongo --eval "db.version()" succeeds before running ceilometer db sync.
(In reply to Alan Pevec from comment #5) > Workaround in puppet-ceilometer could be to loop until mongo --eval > "db.version()" succeeds before running ceilometer db sync. There is no such workaround for HA or ceilometer db sync. The script needs to return the correct status.
Created attachment 873213 [details] Fix.
Created attachment 873215 [details] Updated fix (small tweaks for leading whitespace)
I presume these are for merging back to Fedora/EPEL too?
@pbrady Yeah, it should probably merged back upstream too.
*** Bug 1069798 has been marked as a duplicate of this bug. ***
It sounds like my patch broke a case where an IP wasn't set in the config file. This will have to be conditionalized.
My initscript assumes presence of mongodb in order to operate (using the mongostat command). Thus, we can either make the server package require 'mongodb' or conditionalize the initscript on the presence of the mongostat command.
I don't think making the client package optional would allow the bug to be fixed. For now, I'll rebuild and mongodb-server will require mongodb (but not the other way around).
This didn't come up with 2.2.4 because it already had the fully-versioned requirement on mongodb-server requiring mongodb.
Issue reproduced with mongodb-2.4.6-2.el6ost.x86_64. When attempted to deploy "controller" (Controller with : neutron or nova-network) using foreman Havana A4 2014-04-15.5 While running puppet agent -t -v on the client machine --> getting the following error and the client remain in Error after deployment: The Results: ------------- Foreman-client will remain in status Error 'E' in foreman-server-GUI (*until running the puppet for the second time*) /var/log/messages : -------------------- Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]/ensure) created Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Service[ceilometer-collector] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Service[ceilometer-agent-central] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Exec[ceilometer-dbsync] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Exec[ceilometer-dbsync] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Ceilometer_config[database/connection]) Scheduling refresh of Service[ceilometer-api] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.673 4472 INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 CRITICAL ceilometer [-] could not connect to localhost:27017: [Errno 111] ECONNREFUSED Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer Traceback (most recent call last): Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/bin/ceilometer-dbsync", line 10, in <module> Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer sys.exit(dbsync()) Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 139, in dbsync Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer get_connection(cfg.CONF).upgrade() Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/__init__.py", line 81, in get_connection Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer return get_engine(conf).get_connection(conf) Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 78, in get_connection Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer return Connection(conf) Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 351, in __init__ Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer self.conn = self.CONNECTION_POOL.connect(url) Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/storage/impl_mongodb.py", line 164, in connect Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer safe=True) Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer File "/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py", line 337, in __init__ Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer raise ConnectionFailure(str(e)) Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer ConnectionFailure: could not connect to localhost:27017: [Errno 111] ECONNREFUSED Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.688 4472 TRACE ceilometer Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]) Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]) ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] Apr 22 16:45:45 puma01 yum[4480]: Installed: openstack-ceilometer-collector-2013.2.3-1.el6ost.noarch Apr 22 16:45:46 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Collector/Package[ceilometer-collector]/ensure) created
@Omri Are there any logs from mongodb's installation?
Created attachment 888844 [details] mongodb.log Flavio : I've Added mongodb.log (attached)
That looks like missing dependency in ceilo puppet recipe, Ceilometer::Db starts at 16:45 while mongodb is started only at 16:50 ?
Yes, that looks right, Alan. The mongodb logs indicate that mongodb has started - this out-of-order problem isn't related to the initscript changes.
Alan beat me on this. It sounds like a possible cause. Omri, are these logs taken from the first or second execution? I think a stronger dependency on mongodb is required in the manifest. AFAICS, the mongodb manifest is put in the list of manifests before ceilo's but that doesn't seem to be enough.
(In reply to Flavio Percoco from comment #24) > Omri, are these logs taken from the first or second execution? It took from the same run of https://bugzilla.redhat.com/show_bug.cgi?id=1066408#c19 Just in case - I'll attach the full messages.
Created attachment 888909 [details] messages
Steps to reproduce: -------------------- 1) Install foreman-server using foreman_server.sh 2) Register foreman-client against foreman-server (using foreman_client.sh) 3) Associate the Client with Controller host group (can be either neutron or nova-network) 4) run puppet agent on the client machine (puppet agent -t -v) Results: --------- - Error display during puppet agent running: " Ceilometer::Db/Exec[ceilometer-dbsync]) Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]:" - The client will remain in Error - 'E' in the foreman-server GUI. Workaround: ------------ Run 'puppet agent -t -v' - for the second time.
Looking at the quickstack manifest [1] and Havana packstack template [2], both have mongodb(::server) requirement on ceilometer::db (where sync-db happens), so it should be ok from Puppet point of view. But unfortunately from Omri's log [3] I found that mongodb is started after ceilometer dbsync. Not sure why Puppet ignores require. Will have to investigate that. [1] https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/ceilometer_controller.pp [2] https://github.com/stackforge/packstack/blob/havana/packstack/puppet/templates/ceilometer.pp [3] Apr 22 16:45:42 puma01 puppet-agent[3736]: (/Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]/returns) 2014-04-22 16:45:42.673 4472 INFO ceilometer.storage.impl_mongodb [-] Connecting to MongoDB on [('localhost', 27017)] Apr 22 16:50:41 puma01 puppet-agent[3736]: (/Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure) change from stopped to running failed: Could not start Service[mongodb]: Execution of '/sbin/service mongod start' returned 1:
I wasn't able to reproduce it, but I modified quickstack manifest to be the same regarding MongoDB/Ceilometer dependency and created a pull request: https://github.com/redhat-openstack/astapor/pull/191 Please check it out, test it and let Omri some rpm so he can try if it helped.
Moving this to A5. The upstream patch looks to have some complications that we need to look through before we can pull this into a release.
One issue was during foreman install, the timeout was too short during initial startup with smallfiles turned on. This was resolved in the -4 package.
Verified :(with Puddle: 2014-05-22.1 ) mongodb-2.4.6-4.el6ost.x86_64 openstack-foreman-installer-1.0.12-1.el6ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0577.html