Bug 1036207

Cloning to RHOS, I think for PackStack use we should default to deploying Mongo with small-files enabled. Manual/Foreman configuration should result in "normal configuration" and I am cloning to documentation to note the disk space requirement.

+++ This bug was initially created as a clone of Bug #1028690 +++

Description of problem:
when running packstack and ceilometer is enabled in the answer file. it fails the first time. i have to run it a 2nd time for it to be succesful.

the error message is ERROR : Error appeared during Puppet run: 192.168.122.254_ceilometer.pp
Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]

did this on 2 fresh installs

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Mohammed Arafa on 2013-11-09 11:07:28 EST ---



--- Additional comment from Rushil Chugh on 2013-11-20 12:16:39 EST ---

Hi
I had a few queries about the setup:
Did you run it in all-in-one configuration? Was the configuration setup on Fedora or RHEL?
Rushil

--- Additional comment from Rushil Chugh on 2013-11-20 12:44:57 EST ---

Hi 
Can you also please attach the error log file with this bug? I tried running the same on an all in one file and it worked fine for me.
This happens sometimes when sync_db is run before MongoDB service starts. First time MongoDB starts it's slower because of DB initialization.
Rushil

--- Additional comment from Mohammed Arafa on 2013-11-20 15:07:11 EST ---

rushil

i can reproduce it at will. however, i _JUST_ finally managed to get my openstack instace working and do not want to delete it just yet.

so to reproduce it this is my packstack answer file edits:
https://raw.github.com/marafa/openstack/master/openstack-notes.sh 

in particular:
#####modify ~/packstack.answer
sed -i 's/CONFIG_NTP_SERVERS=/CONFIG_NTP_SERVERS=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org/g' ~/packstack.answer
sed -i 's/CONFIG_HORIZON_SSL=n/CONFIG_HORIZON_SSL=y/g' ~/packstack.answer 
sed -i 's/PW=.*/PW=password/g' ~/packstack.answer 
sed -i 's/CONFIG_SWIFT_INSTALL=n/CONFIG_SWIFT_INSTALL=y/g' ~/packstack.answer
sed -i 's/CONFIG_CINDER_VOLUMES_SIZE=20G/CONFIG_CINDER_VOLUMES_SIZE=5G/g' ~/packstack.answer
sed -i 's,CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.4.0/22,CONFIG_NOVA_NETWORK_FLOATRANGE=192.168.0.0/24,g' ~/packstack.answer
sed -i 's/CONFIG_PROVISION_DEMO=n/CONFIG_PROVISION_DEMO=y/g' ~/packstack.answer
sed -i 's,CONFIG_PROVISION_DEMO_FLOATRANGE=172.24.4.224/28,CONFIG_PROVISION_DEMO_FLOATRANGE=192.168.0.0/24,g' ~/packstack.answer

####vlan support
sed -i 's/CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=local/CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vlan/g' ~/packstack.answer
sed -i 's/CONFIG_NEUTRON_OVS_VLAN_RANGES=/CONFIG_NEUTRON_OVS_VLAN_RANGES=physnet1:1000:2999/g' ~/packstack.answer
sed -i 's/CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=/CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=physnet1:br-eth0:1/g' ~/packstack.answer


and it was on centos6.4 fully patched before installing packstack

--- Additional comment from Mohammed Arafa on 2013-11-20 15:07:48 EST ---

hi again
i see i attached the packstack answer file previously

thanks

--- Additional comment from Mohammed Arafa on 2013-11-21 21:57:21 EST ---

ERROR : Error appeared during Puppet run: 192.168.122.254_ceilometer.pp
Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]

[root@openstack ~]# packstack --allinone --nagios-install=n

i changed the parameters of packstack and yet i got the same issue. fresh install

--- Additional comment from Eoghan Glynn on 2013-11-22 04:22:30 EST ---

Hi Mohammed,

Can you also attach your /var/log/mongodb/mongodb.log?

(specifically the log fragment from around the time that the problem was encountered)

Thanks,
Eoghan

--- Additional comment from Mohammed Arafa on 2013-11-22 06:49:48 EST ---

[root@openstack mongodb]# cat mongodb.log
Fri Nov 22 06:45:15.320 [initandlisten] MongoDB starting : pid=2146 port=27017 dbpath=/var/lib/mongodb 64-bit host=openstack.marafa.vm
Fri Nov 22 06:45:15.321 [initandlisten] db version v2.4.6
Fri Nov 22 06:45:15.321 [initandlisten] git version: nogitversion
Fri Nov 22 06:45:15.321 [initandlisten] build info: Linux buildvm-09.phx2.fedoraproject.org 3.10.9-200.fc19.x86_64 #1 SMP Wed Aug 21 19:27:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_41
Fri Nov 22 06:45:15.321 [initandlisten] allocator: tcmalloc
Fri Nov 22 06:45:15.321 [initandlisten] options: { bind_ip: "127.0.0.1", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", fork: "true", journal: "true", logpath: "/var/log/mongodb/mongodb.log", nohttpinterface: "true", pidfilepath: "/var/run/mongodb/mongodb.pid", port: 27017, quiet: true }
Fri Nov 22 06:45:15.323 [initandlisten] 
Fri Nov 22 06:45:15.323 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 512KB
Fri Nov 22 06:45:15.323 [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
Fri Nov 22 06:45:15.323 [initandlisten] **          http://dochub.mongodb.org/core/readahead
Fri Nov 22 06:45:15.344 [initandlisten] journal dir=/var/lib/mongodb/journal
Fri Nov 22 06:45:15.345 [initandlisten] recover : no journal files present, no recovery needed
Fri Nov 22 06:45:15.345 [initandlisten] 
Fri Nov 22 06:45:15.345 [initandlisten] ERROR: Insufficient free space for journal files
Fri Nov 22 06:45:15.345 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles
Fri Nov 22 06:45:15.345 [initandlisten] 
Fri Nov 22 06:45:15.345 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Fri Nov 22 06:45:15.345 dbexit: 
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to close listening sockets...
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to flush diaglog...
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: going to close sockets...
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: waiting for fs preallocator...
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: lock for final commit...
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: final commit...
Fri Nov 22 06:45:15.345 [initandlisten] shutdown: closing all files...
Fri Nov 22 06:45:15.345 [initandlisten] closeAllFiles() finished
Fri Nov 22 06:45:15.345 [initandlisten] journalCleanup...
Fri Nov 22 06:45:15.345 [initandlisten] removeJournalFiles
Fri Nov 22 06:45:15.354 [initandlisten] shutdown: removing fs lock...
Fri Nov 22 06:45:15.354 dbexit: really exiting now
[root@openstack mongodb]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_openstack-lv_root
                       3874440   1768316   1909312  49% /
tmpfs                  4029800         0   4029800   0% /dev/shm
/dev/vda1               495844     87373    382871  19% /boot
/srv/loopback-device/device1
                        982936     34056    896452   4% /srv/node/device1
[root@openstack mongodb]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_openstack-lv_root
                      3.7G  1.7G  1.9G  49% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/vda1             485M   86M  374M  19% /boot
/srv/loopback-device/device1
                      960M   34M  876M   4% /srv/node/device1

--- Additional comment from Mohammed Arafa on 2013-11-22 06:51:27 EST ---



--- Additional comment from Mohammed Arafa on 2013-11-22 06:51:57 EST ---



--- Additional comment from Mohammed Arafa on 2013-11-22 06:52:18 EST ---



--- Additional comment from Eoghan Glynn on 2013-11-22 07:34:45 EST ---

Hi Mohammed,

So here's your problem I think:

  ERROR: Insufficient free space for journal files

Note that mongodb by default eagerly preallocates journal files.

Are you installing openstack in a highly resource-constrained environment, such as on a small-flavor VM without significant disk space?

The time taken by mongodb startup in such an environment may also be problematic.

You can work-around these issues by pre-installing mongo as follows, setting the --smallfiles option to reduce the initial file size to 512Mb, and also increase the startup timeout:

   # pre-install mongodb
   sudo yum install -y mongodb-server mongodb

   # use smaller initial file size
   sudo sed -i '/--smallfiles/!s/OPTIONS=\"/OPTIONS=\"--smallfiles /' /etc/sysconfig/mongod

   # increase the start-up timeout
   sudo sed -i '/^\[Service\]$/ a\
     TimeoutStartSec=360' /usr/lib/systemd/system/mongod.service

   # fire up the mongodb service initially to force the journal pre-allocation
   sudo service mongod start
   sudo service mongod status
   sudo service mongod stop

Cheers,
Eoghan

--- Additional comment from Mohammed Arafa on 2013-11-24 11:12:31 EST ---

so ..the journal wanted 3.x GB of free space is required?
or else use --smallfiles ?

the logic of it escapes me.

however, i have another question now, how will having mongodb preinstalled (most likely with a password) affect the packstack installation process?

--- Additional comment from Eoghan Glynn on 2013-11-25 04:29:26 EST ---


Hi Mohammed,

The logic of mongodb journal file pre-allocation is an aspect of mongodb that's outside our control, other than via the config options referred to above.

Preinstallation of mongodb will not cause a problem for packstack.

Given the above information, are you happy for me to close off this issue?

(You would of course be welcome to file another bug if you other related issue crops up for you).

Cheers,
Eoghan

--- Additional comment from Mohammed Arafa on 2013-11-25 13:22:46 EST ---

thanks eoghan

--- Additional comment from Stephen Gordon on 2013-11-29 13:38:34 EST ---

I'm re-opening this, I think we should default to --smallfiles to alleviate this somewhat - I'm also cloning to RHOS and the associated documentation components to ensure we note this in the up front requirements.