+++ This bug was initially created as a clone of Bug #1036207 +++ Cloning to RHOS, I think for PackStack use we should default to deploying Mongo with small-files enabled. Manual/Foreman configuration should result in "normal configuration" and I am cloning to documentation to note the disk space requirement. +++ This bug was initially created as a clone of Bug #1028690 +++ Description of problem: when running packstack and ceilometer is enabled in the answer file. it fails the first time. i have to run it a 2nd time for it to be succesful. the error message is ERROR : Error appeared during Puppet run: 192.168.122.254_ceilometer.pp Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] did this on 2 fresh installs Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Mohammed Arafa on 2013-11-09 11:07:28 EST --- --- Additional comment from Rushil Chugh on 2013-11-20 12:16:39 EST --- Hi I had a few queries about the setup: Did you run it in all-in-one configuration? Was the configuration setup on Fedora or RHEL? Rushil --- Additional comment from Rushil Chugh on 2013-11-20 12:44:57 EST --- Hi Can you also please attach the error log file with this bug? I tried running the same on an all in one file and it worked fine for me. This happens sometimes when sync_db is run before MongoDB service starts. First time MongoDB starts it's slower because of DB initialization. Rushil --- Additional comment from Mohammed Arafa on 2013-11-20 15:07:11 EST --- rushil i can reproduce it at will. however, i _JUST_ finally managed to get my openstack instace working and do not want to delete it just yet. so to reproduce it this is my packstack answer file edits: https://raw.github.com/marafa/openstack/master/openstack-notes.sh in particular: #####modify ~/packstack.answer sed -i 's/CONFIG_NTP_SERVERS=/CONFIG_NTP_SERVERS=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org/g' ~/packstack.answer sed -i 's/CONFIG_HORIZON_SSL=n/CONFIG_HORIZON_SSL=y/g' ~/packstack.answer sed -i 's/PW=.*/PW=password/g' ~/packstack.answer sed -i 's/CONFIG_SWIFT_INSTALL=n/CONFIG_SWIFT_INSTALL=y/g' ~/packstack.answer sed -i 's/CONFIG_CINDER_VOLUMES_SIZE=20G/CONFIG_CINDER_VOLUMES_SIZE=5G/g' ~/packstack.answer sed -i 's,CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.4.0/22,CONFIG_NOVA_NETWORK_FLOATRANGE=192.168.0.0/24,g' ~/packstack.answer sed -i 's/CONFIG_PROVISION_DEMO=n/CONFIG_PROVISION_DEMO=y/g' ~/packstack.answer sed -i 's,CONFIG_PROVISION_DEMO_FLOATRANGE=172.24.4.224/28,CONFIG_PROVISION_DEMO_FLOATRANGE=192.168.0.0/24,g' ~/packstack.answer ####vlan support sed -i 's/CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=local/CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vlan/g' ~/packstack.answer sed -i 's/CONFIG_NEUTRON_OVS_VLAN_RANGES=/CONFIG_NEUTRON_OVS_VLAN_RANGES=physnet1:1000:2999/g' ~/packstack.answer sed -i 's/CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=/CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-eth0:1/g' ~/packstack.answer and it was on centos6.4 fully patched before installing packstack --- Additional comment from Mohammed Arafa on 2013-11-20 15:07:48 EST --- hi again i see i attached the packstack answer file previously thanks --- Additional comment from Mohammed Arafa on 2013-11-21 21:57:21 EST --- ERROR : Error appeared during Puppet run: 192.168.122.254_ceilometer.pp Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] [root@openstack ~]# packstack --allinone --nagios-install=n i changed the parameters of packstack and yet i got the same issue. fresh install --- Additional comment from Eoghan Glynn on 2013-11-22 04:22:30 EST --- Hi Mohammed, Can you also attach your /var/log/mongodb/mongodb.log? (specifically the log fragment from around the time that the problem was encountered) Thanks, Eoghan --- Additional comment from Mohammed Arafa on 2013-11-22 06:49:48 EST --- [root@openstack mongodb]# cat mongodb.log Fri Nov 22 06:45:15.320 [initandlisten] MongoDB starting : pid=2146 port=27017 dbpath=/var/lib/mongodb 64-bit host=openstack.marafa.vm Fri Nov 22 06:45:15.321 [initandlisten] db version v2.4.6 Fri Nov 22 06:45:15.321 [initandlisten] git version: nogitversion Fri Nov 22 06:45:15.321 [initandlisten] build info: Linux buildvm-09.phx2.fedoraproject.org 3.10.9-200.fc19.x86_64 #1 SMP Wed Aug 21 19:27:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_41 Fri Nov 22 06:45:15.321 [initandlisten] allocator: tcmalloc Fri Nov 22 06:45:15.321 [initandlisten] options: { bind_ip: "127.0.0.1", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", fork: "true", journal: "true", logpath: "/var/log/mongodb/mongodb.log", nohttpinterface: "true", pidfilepath: "/var/run/mongodb/mongodb.pid", port: 27017, quiet: true } Fri Nov 22 06:45:15.323 [initandlisten] Fri Nov 22 06:45:15.323 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 512KB Fri Nov 22 06:45:15.323 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less Fri Nov 22 06:45:15.323 [initandlisten] ** http://dochub.mongodb.org/core/readahead Fri Nov 22 06:45:15.344 [initandlisten] journal dir=/var/lib/mongodb/journal Fri Nov 22 06:45:15.345 [initandlisten] recover : no journal files present, no recovery needed Fri Nov 22 06:45:15.345 [initandlisten] Fri Nov 22 06:45:15.345 [initandlisten] ERROR: Insufficient free space for journal files Fri Nov 22 06:45:15.345 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles Fri Nov 22 06:45:15.345 [initandlisten] Fri Nov 22 06:45:15.345 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating Fri Nov 22 06:45:15.345 dbexit: Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to close listening sockets... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to flush diaglog... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to close sockets... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: waiting for fs preallocator... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: lock for final commit... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: final commit... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: closing all files... Fri Nov 22 06:45:15.345 [initandlisten] closeAllFiles() finished Fri Nov 22 06:45:15.345 [initandlisten] journalCleanup... Fri Nov 22 06:45:15.345 [initandlisten] removeJournalFiles Fri Nov 22 06:45:15.354 [initandlisten] shutdown: removing fs lock... Fri Nov 22 06:45:15.354 dbexit: really exiting now [root@openstack mongodb]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_openstack-lv_root 3874440 1768316 1909312 49% / tmpfs 4029800 0 4029800 0% /dev/shm /dev/vda1 495844 87373 382871 19% /boot /srv/loopback-device/device1 982936 34056 896452 4% /srv/node/device1 [root@openstack mongodb]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_openstack-lv_root 3.7G 1.7G 1.9G 49% / tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/vda1 485M 86M 374M 19% /boot /srv/loopback-device/device1 960M 34M 876M 4% /srv/node/device1 --- Additional comment from Mohammed Arafa on 2013-11-22 06:51:27 EST --- --- Additional comment from Mohammed Arafa on 2013-11-22 06:51:57 EST --- --- Additional comment from Mohammed Arafa on 2013-11-22 06:52:18 EST --- --- Additional comment from Eoghan Glynn on 2013-11-22 07:34:45 EST --- Hi Mohammed, So here's your problem I think: ERROR: Insufficient free space for journal files Note that mongodb by default eagerly preallocates journal files. Are you installing openstack in a highly resource-constrained environment, such as on a small-flavor VM without significant disk space? The time taken by mongodb startup in such an environment may also be problematic. You can work-around these issues by pre-installing mongo as follows, setting the --smallfiles option to reduce the initial file size to 512Mb, and also increase the startup timeout: # pre-install mongodb sudo yum install -y mongodb-server mongodb # use smaller initial file size sudo sed -i '/--smallfiles/!s/OPTIONS=\"/OPTIONS=\"--smallfiles /' /etc/sysconfig/mongod # increase the start-up timeout sudo sed -i '/^\[Service\]$/ a\ TimeoutStartSec=360' /usr/lib/systemd/system/mongod.service # fire up the mongodb service initially to force the journal pre-allocation sudo service mongod start sudo service mongod status sudo service mongod stop Cheers, Eoghan --- Additional comment from Mohammed Arafa on 2013-11-24 11:12:31 EST --- so ..the journal wanted 3.x GB of free space is required? or else use --smallfiles ? the logic of it escapes me. however, i have another question now, how will having mongodb preinstalled (most likely with a password) affect the packstack installation process? --- Additional comment from Eoghan Glynn on 2013-11-25 04:29:26 EST --- Hi Mohammed, The logic of mongodb journal file pre-allocation is an aspect of mongodb that's outside our control, other than via the config options referred to above. Preinstallation of mongodb will not cause a problem for packstack. Given the above information, are you happy for me to close off this issue? (You would of course be welcome to file another bug if you other related issue crops up for you). Cheers, Eoghan --- Additional comment from Mohammed Arafa on 2013-11-25 13:22:46 EST --- thanks eoghan --- Additional comment from Stephen Gordon on 2013-11-29 13:38:34 EST --- I'm re-opening this, I think we should default to --smallfiles to alleviate this somewhat - I'm also cloning to RHOS and the associated documentation components to ensure we note this in the up front requirements. --- Additional comment from Lon Hohberger on 2014-01-20 11:28:12 EST --- devel_ack+ for bug in POST --- Additional comment from Ami Jeain on 2014-02-09 04:55:07 EST --- no capacity to test Ceilometer in Havana --- Additional comment from errata-xmlrpc on 2014-02-13 15:50:03 EST --- Bug report changed to ON_QA status by Errata System. A QE request has been submitted for advisory RHBA-2014:17057-01 https://errata.devel.redhat.com/advisory/17057 --- Additional comment from Attila Fazekas on 2014-02-17 06:24:23 EST --- With small files (openstack-packstack-2013.2.1-0.23.dev979.el6ost): du -sh /var/lib/mongodb/ 65M /var/lib/mongodb/ Without small files (openstack-packstack-2013.2.1-0.22.dev956.el6ost): du -sh /var/lib/mongodb/ 209M /var/lib/mongodb/ I have doubts is it wise to reduce the file-sizes for bigger scale installation, but may be it is ok for small scale test installation. --- Additional comment from Mohammed Arafa on 2014-02-17 08:29:04 EST --- @attila isnt this a packstack switch? --- Additional comment from Francesco Vollero on 2014-02-17 09:00:33 EST --- Mohammed, no isn't a switch. Packstack it's meant to be done on small environments. A switch could be done on a future release, but is a design feature. --- Additional comment from errata-xmlrpc on 2014-02-21 16:28:34 EST --- This bug has been dropped from advisory RHBA-2014:17057 by Lon Hohberger (lhh) --- Additional comment from Lon Hohberger on 2014-02-21 16:38:07 EST --- This bug is now part of the following RHSA: https://errata.devel.redhat.com/advisory/17118
So, according to Francesco, there is a race condition, wherein mongo doesnt always start in a timely manner, resulting in systemd killing it. Restart is usually successful. His patch for packstack (linked) updates to use 'smallfiles' flag from mongo, and also changes use of mongo ->mongo::server in the puppet calls. Foreman likely needs the same changes in quickstack, as this seems to have fixed it on the packstack side. Not really sure why the underlying puppet module doesn't just work by default, but that is another issue.
Moving to mongodb.
This has a high impact on all foreman installs, with likely more complications around the HA ones.
I was pretty sure this was due to a bug in the initscript.
I'm not happy with shipping mongodb with `smallfiles = true`. This is useful when there are *many* databases with *small* amount of data each. It reduces the size of journal files in order to avoid locking so much space ahead. This is not RHOS case, though. I agree it's useful for development and testing but I don't think it's ideal for production environments. At least not with the throughput we expect to have in Ceilometer. I'd recommend setting this from foreman / puppet if needed.
Personally I don't think we want to depend on smallfiles = true to make this work. That is probably a reasonable solution for packstack, but I can't imagine we should be doing that in larger installations. I was hoping we could either confirm/deny that the initscript fully addressed the normal case. We can try retesting via foreman to confirm whether or not the behavior still exists but I don't think it is the right thing to do to have foreman set smallfiles = true if the condition still exists.
@Mike, I fully agree! I don't think depending on `smallfiles = true` to make this work is the right thing to do. In other places (devstack) we poll from mongodb (or ceilomenter) and wait until it's up. Does this sound reasonable for the foreman case? As for mongodb's initscript. It currently addresses the normal case, which is smallfiles = false, and as mentioned in the my other comment I think it should stay that way. Thanks Mike!
After reading this bug again, it looks a lot like #1066408 The fix proposed for that bug should also fix this one here. The fix makes the initscript wait for mongod to be ready to accept new connections, which makes the need of using smallfiles to go away. I'm closing this bug as a duplicate of #1066408 feel free to open if you think otherwise. *** This bug has been marked as a duplicate of bug 1066408 ***
This is indeed now fixed, with a combination of the above BZ and a fix on our side to ensure proper ordering