Bug 1220250

Summary:	poodle 2015-05-06.12 - packstack failed - mongodb error
Product:	Red Hat OpenStack	Reporter:	Roey Dekel <rdekel>
Component:	openstack-packstack	Assignee:	Martin Magr <mmagr>
Status:	CLOSED ERRATA	QA Contact:	Ido Ovadia <iovadia>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	aortega, ekuris, oblaut, sclewis, yeylon, yprokule
Target Milestone:	ga
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-packstack-2015.1-0.2.dev1537.gba5183c.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-08-05 13:23:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Roey Dekel 2015-05-11 05:45:15 UTC

Description of problem:
packstack failed

Version-Release number of selected component (if applicable):
poodle: 2015-05-06.12
openstack-packstack-puppet-2015.1-0.1.dev1537.gba5183c.el7ost.noarch
openstack-packstack-2015.1-0.1.dev1537.gba5183c.el7ost.noarch
mongodb-server-2.6.9-1.el7ost.x86_64

How reproducible:
Tried AIO and distibuted nested (BLUE), both failed

Steps to Reproduce:
1. Install packstack with:
   # rhos-release -d 7 -p 2015-05-06.12

Actual results:
---------------
10.35.160.37_mongodb.pp:                          [ ERROR ]        
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.35.160.37_mongodb.pp
Error: Unable to connect to mongodb server! (10.35.160.37:27017)
You will find full trace in log /var/tmp/packstack/20150510-162712-K4OJrg/manifests/10.35.160.37_mongodb.pp.log
Please check log file /var/tmp/packstack/20150510-162712-K4OJrg/openstack-setup.log for more information
Additional information:
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * Warning: NetworkManager is active on 10.35.160.37. OpenStack networking currently does not work on systems that have the Network Manager service enabled.
 * File /root/keystonerc_admin has been created on OpenStack client host 10.35.160.37. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://10.35.160.37/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * To use Nagios, browse to http://10.35.160.37/nagios username: nagiosadmin, password: b45052b4e6394d9d

Additional info:
----------------
[root@puma14 ~]# ps -elf | grep mongodb
1 S mongodb   4672     1  0  80   0 - 128426 poll_s May10 ?       00:02:04 /usr/bin/mongod --quiet -f /etc/mongod.conf run
0 S root     13413 13246  0  80   0 - 28161 pipe_w 08:42 pts/1    00:00:00 grep --color=auto mongodb

[root@puma14 ~]# iptables -L | grep 27017
ACCEPT     tcp  --  puma14.scl.lab.tlv.redhat.com  anywhere             multiport dports 27017 /* 001 mongodb-server incoming mongodb_server */

answer file:    http://pastebin.test.redhat.com/282049
full trace:     http://pastebin.test.redhat.com/282050
log file:       http://pastebin.test.redhat.com/282051

Comment 3 Roey Dekel 2015-05-11 07:19:40 UTC

Might be related to https://bugzilla.redhat.com/show_bug.cgi?id=1212174

Comment 4 Yurii Prokulevych 2015-05-11 11:52:58 UTC

My 2cents since I've also encountered this problem.

As a workaround I've changed 'bind_ip' within /etc/mongod.conf to point to my IP address, restarted service and re-run installation.

After some more digging it was found that config file used during Mongo start is passed via $OPTIONS argument that actually points to /etc/sysconfig/mongod: 

cat /usr/lib/systemd/system/mongod.service
[Unit]
Description=High-performance, schema-free document-oriented database
After=syslog.target network.target

[Service]
Type=forking
User=mongodb
EnvironmentFile=/etc/sysconfig/mongod
#PIDFile=${PIDFILE-/var/run/mongodb/mongod.pid}  # my comment.
PIDFile=/var/run/mongodb/mongod/pid
ExecStart=/usr/bin/mongod $OPTIONS run
PrivateTmp=true
LimitNOFILE=64000
TimeoutStartSec=180

[Install]
WantedBy=multi-user.target

#################
cat /etc/sysconfig/mongod
OPTIONS="--quiet -f /etc/mongod.conf"

So, theoretically changing /etc/sysconfig/mongod to use correct/needed config file has to fix the issue. (honestly haven't tried this)

Also the problem is that /etc/mongodb.conf is generated by packstack and hence mongod.service unit doesn't know anything about it:
rpm -qf /etc/mongodb.conf 
file /etc/mongodb.conf is not owned by any package

rpm -qf /etc/mongod.conf 
mongodb-server-2.6.9-1.el7ost.x86_64

Some info about installed packages:

python-ceilometerclient-1.0.13-1.el7ost.noarch
openstack-ceilometer-alarm-2015.1.0-1.el7ost.noarch
python-ceilometer-2015.1.0-1.el7ost.noarch
openstack-ceilometer-notification-2015.1.0-1.el7ost.noarch
openstack-ceilometer-central-2015.1.0-1.el7ost.noarch
openstack-ceilometer-api-2015.1.0-1.el7ost.noarch
openstack-ceilometer-compute-2015.1.0-1.el7ost.noarch
openstack-ceilometer-collector-2015.1.0-1.el7ost.noarch
openstack-ceilometer-common-2015.1.0-1.el7ost.noarch

openstack-packstack-puppet-2015.1-0.1.dev1537.gba5183c.el7ost.noarch
openstack-packstack-2015.1-0.1.dev1537.gba5183c.el7ost.noarch

python-pymongo-2.5.2-2.el7ost.x86_64
mongodb-server-2.6.9-1.el7ost.x86_64

---
Yurii

Comment 6 Eran Kuris 2015-05-14 11:20:07 UTC

Run the installation with packstack . ceilometer enabled . 
the installation pass

Comment 10 errata-xmlrpc 2015-08-05 13:23:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1548