Cloning to RHOS, I think for PackStack use we should default to deploying Mongo with small-files enabled. Manual/Foreman configuration should result in "normal configuration" and I am cloning to documentation to note the disk space requirement. +++ This bug was initially created as a clone of Bug #1028690 +++ Description of problem: when running packstack and ceilometer is enabled in the answer file. it fails the first time. i have to run it a 2nd time for it to be succesful. the error message is ERROR : Error appeared during Puppet run: 192.168.122.254_ceilometer.pp Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] did this on 2 fresh installs Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Mohammed Arafa on 2013-11-09 11:07:28 EST --- --- Additional comment from Rushil Chugh on 2013-11-20 12:16:39 EST --- Hi I had a few queries about the setup: Did you run it in all-in-one configuration? Was the configuration setup on Fedora or RHEL? Rushil --- Additional comment from Rushil Chugh on 2013-11-20 12:44:57 EST --- Hi Can you also please attach the error log file with this bug? I tried running the same on an all in one file and it worked fine for me. This happens sometimes when sync_db is run before MongoDB service starts. First time MongoDB starts it's slower because of DB initialization. Rushil --- Additional comment from Mohammed Arafa on 2013-11-20 15:07:11 EST --- rushil i can reproduce it at will. however, i _JUST_ finally managed to get my openstack instace working and do not want to delete it just yet. so to reproduce it this is my packstack answer file edits: https://raw.github.com/marafa/openstack/master/openstack-notes.sh in particular: #####modify ~/packstack.answer sed -i 's/CONFIG_NTP_SERVERS=/CONFIG_NTP_SERVERS=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org/g' ~/packstack.answer sed -i 's/CONFIG_HORIZON_SSL=n/CONFIG_HORIZON_SSL=y/g' ~/packstack.answer sed -i 's/PW=.*/PW=password/g' ~/packstack.answer sed -i 's/CONFIG_SWIFT_INSTALL=n/CONFIG_SWIFT_INSTALL=y/g' ~/packstack.answer sed -i 's/CONFIG_CINDER_VOLUMES_SIZE=20G/CONFIG_CINDER_VOLUMES_SIZE=5G/g' ~/packstack.answer sed -i 's,CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.4.0/22,CONFIG_NOVA_NETWORK_FLOATRANGE=192.168.0.0/24,g' ~/packstack.answer sed -i 's/CONFIG_PROVISION_DEMO=n/CONFIG_PROVISION_DEMO=y/g' ~/packstack.answer sed -i 's,CONFIG_PROVISION_DEMO_FLOATRANGE=172.24.4.224/28,CONFIG_PROVISION_DEMO_FLOATRANGE=192.168.0.0/24,g' ~/packstack.answer ####vlan support sed -i 's/CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=local/CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vlan/g' ~/packstack.answer sed -i 's/CONFIG_NEUTRON_OVS_VLAN_RANGES=/CONFIG_NEUTRON_OVS_VLAN_RANGES=physnet1:1000:2999/g' ~/packstack.answer sed -i 's/CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=/CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-eth0:1/g' ~/packstack.answer and it was on centos6.4 fully patched before installing packstack --- Additional comment from Mohammed Arafa on 2013-11-20 15:07:48 EST --- hi again i see i attached the packstack answer file previously thanks --- Additional comment from Mohammed Arafa on 2013-11-21 21:57:21 EST --- ERROR : Error appeared during Puppet run: 192.168.122.254_ceilometer.pp Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0] [root@openstack ~]# packstack --allinone --nagios-install=n i changed the parameters of packstack and yet i got the same issue. fresh install --- Additional comment from Eoghan Glynn on 2013-11-22 04:22:30 EST --- Hi Mohammed, Can you also attach your /var/log/mongodb/mongodb.log? (specifically the log fragment from around the time that the problem was encountered) Thanks, Eoghan --- Additional comment from Mohammed Arafa on 2013-11-22 06:49:48 EST --- [root@openstack mongodb]# cat mongodb.log Fri Nov 22 06:45:15.320 [initandlisten] MongoDB starting : pid=2146 port=27017 dbpath=/var/lib/mongodb 64-bit host=openstack.marafa.vm Fri Nov 22 06:45:15.321 [initandlisten] db version v2.4.6 Fri Nov 22 06:45:15.321 [initandlisten] git version: nogitversion Fri Nov 22 06:45:15.321 [initandlisten] build info: Linux buildvm-09.phx2.fedoraproject.org 3.10.9-200.fc19.x86_64 #1 SMP Wed Aug 21 19:27:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_41 Fri Nov 22 06:45:15.321 [initandlisten] allocator: tcmalloc Fri Nov 22 06:45:15.321 [initandlisten] options: { bind_ip: "127.0.0.1", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", fork: "true", journal: "true", logpath: "/var/log/mongodb/mongodb.log", nohttpinterface: "true", pidfilepath: "/var/run/mongodb/mongodb.pid", port: 27017, quiet: true } Fri Nov 22 06:45:15.323 [initandlisten] Fri Nov 22 06:45:15.323 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 512KB Fri Nov 22 06:45:15.323 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less Fri Nov 22 06:45:15.323 [initandlisten] ** http://dochub.mongodb.org/core/readahead Fri Nov 22 06:45:15.344 [initandlisten] journal dir=/var/lib/mongodb/journal Fri Nov 22 06:45:15.345 [initandlisten] recover : no journal files present, no recovery needed Fri Nov 22 06:45:15.345 [initandlisten] Fri Nov 22 06:45:15.345 [initandlisten] ERROR: Insufficient free space for journal files Fri Nov 22 06:45:15.345 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles Fri Nov 22 06:45:15.345 [initandlisten] Fri Nov 22 06:45:15.345 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating Fri Nov 22 06:45:15.345 dbexit: Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to close listening sockets... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to flush diaglog... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to close sockets... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: waiting for fs preallocator... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: lock for final commit... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: final commit... Fri Nov 22 06:45:15.345 [initandlisten] shutdown: closing all files... Fri Nov 22 06:45:15.345 [initandlisten] closeAllFiles() finished Fri Nov 22 06:45:15.345 [initandlisten] journalCleanup... Fri Nov 22 06:45:15.345 [initandlisten] removeJournalFiles Fri Nov 22 06:45:15.354 [initandlisten] shutdown: removing fs lock... Fri Nov 22 06:45:15.354 dbexit: really exiting now [root@openstack mongodb]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_openstack-lv_root 3874440 1768316 1909312 49% / tmpfs 4029800 0 4029800 0% /dev/shm /dev/vda1 495844 87373 382871 19% /boot /srv/loopback-device/device1 982936 34056 896452 4% /srv/node/device1 [root@openstack mongodb]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_openstack-lv_root 3.7G 1.7G 1.9G 49% / tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/vda1 485M 86M 374M 19% /boot /srv/loopback-device/device1 960M 34M 876M 4% /srv/node/device1 --- Additional comment from Mohammed Arafa on 2013-11-22 06:51:27 EST --- --- Additional comment from Mohammed Arafa on 2013-11-22 06:51:57 EST --- --- Additional comment from Mohammed Arafa on 2013-11-22 06:52:18 EST --- --- Additional comment from Eoghan Glynn on 2013-11-22 07:34:45 EST --- Hi Mohammed, So here's your problem I think: ERROR: Insufficient free space for journal files Note that mongodb by default eagerly preallocates journal files. Are you installing openstack in a highly resource-constrained environment, such as on a small-flavor VM without significant disk space? The time taken by mongodb startup in such an environment may also be problematic. You can work-around these issues by pre-installing mongo as follows, setting the --smallfiles option to reduce the initial file size to 512Mb, and also increase the startup timeout: # pre-install mongodb sudo yum install -y mongodb-server mongodb # use smaller initial file size sudo sed -i '/--smallfiles/!s/OPTIONS=\"/OPTIONS=\"--smallfiles /' /etc/sysconfig/mongod # increase the start-up timeout sudo sed -i '/^\[Service\]$/ a\ TimeoutStartSec=360' /usr/lib/systemd/system/mongod.service # fire up the mongodb service initially to force the journal pre-allocation sudo service mongod start sudo service mongod status sudo service mongod stop Cheers, Eoghan --- Additional comment from Mohammed Arafa on 2013-11-24 11:12:31 EST --- so ..the journal wanted 3.x GB of free space is required? or else use --smallfiles ? the logic of it escapes me. however, i have another question now, how will having mongodb preinstalled (most likely with a password) affect the packstack installation process? --- Additional comment from Eoghan Glynn on 2013-11-25 04:29:26 EST --- Hi Mohammed, The logic of mongodb journal file pre-allocation is an aspect of mongodb that's outside our control, other than via the config options referred to above. Preinstallation of mongodb will not cause a problem for packstack. Given the above information, are you happy for me to close off this issue? (You would of course be welcome to file another bug if you other related issue crops up for you). Cheers, Eoghan --- Additional comment from Mohammed Arafa on 2013-11-25 13:22:46 EST --- thanks eoghan --- Additional comment from Stephen Gordon on 2013-11-29 13:38:34 EST --- I'm re-opening this, I think we should default to --smallfiles to alleviate this somewhat - I'm also cloning to RHOS and the associated documentation components to ensure we note this in the up front requirements.
With small files (openstack-packstack-2013.2.1-0.23.dev979.el6ost): du -sh /var/lib/mongodb/ 65M /var/lib/mongodb/ Without small files (openstack-packstack-2013.2.1-0.22.dev956.el6ost): du -sh /var/lib/mongodb/ 209M /var/lib/mongodb/ I have doubts is it wise to reduce the file-sizes for bigger scale installation, but may be it is ok for small scale test installation.
@attila isnt this a packstack switch?
Mohammed, no isn't a switch. Packstack it's meant to be done on small environments. A switch could be done on a future release, but is a design feature.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2014-0233.html