Bug 1098821

Summary: Packstack allinone installation fails due to failure to start rabbitmq-server during amqp.pp on CentOS 6.5
Product: [Community] RDO Reporter: Bill Bruns <billbruns>
Component: openstack-packstackAssignee: Martin Magr <mmagr>
Status: CLOSED CURRENTRELEASE QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: aortega, derekh, lbezdick, mmagr, mruckman, msuchy, srevivo, yeylon
Target Milestone: ---   
Target Release: Icehouse   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-packstack-2014.1.1-0.31.4.dev1279 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-30 23:04:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bill Bruns 2014-05-18 22:51:20 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:
Install Packstack on CentOS 6.5 on an Intel i3 with 8 GB ram and 1 TB disk

Steps to Reproduce:
1. Freshly install CentOS 6.5 using the Basic Server flavor
2. As root, update yum and install Packstack from http://openstack.redhat.com/Quickstart
3. At the very end it fails

Actual results:
It fails at the end while applying the Puppt file amqp.pp

Expected results:
Successful installation

Additional info:
Here are the messages at the end of the installation:
Applying 172.16.0.19_amqp.pp
172.16.0.19_mysql.pp:                                [ DONE ]
172.16.0.19_amqp.pp:                              [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 172.16.0.19_amqp.pp
Error: Could not start Service[rabbitmq-server]: Execution of '/sbin/service rabbitmq-server start' returned 1: Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/startup_{log, _err}
You will find full trace in log /var/tmp/packstack/20140517-192149-UFbaRi/manifests/172.16.0.19_amqp.pp.log
Please check log file /var/tmp/packstack/20140517-192149-UFbaRi/openstack-setup.log for more information

May be similar to other queuing bugs such as Bug 1097306

Here is the log file:
[root@oshost ~]# more /var/tmp/packstack/20140517-192149-UFbaRi/manifests/172.16.0.19_amqp.pp.log
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for oshost.bb.com in environment production in 0.79 seconds
Notice: /Stage[main]/Rabbitmq::Repo::Rhel/Exec[rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc]/returns: executed succe
ssfully
Notice: /Stage[main]/Main/Enable_rabbitmq[rabbitmq]/Package[erlang]/ensure: created
Notice: /Stage[main]/Rabbitmq::Install/Package[rabbitmq-server]/ensure: created
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq.config]/ensure: defined content as '{md5}5647446b04534bab24f4b348af808e93'
Notice: /Stage[main]/Rabbitmq::Config/File[/etc/rabbitmq/ssl]/ensure: created
Notice: /Stage[main]/Rabbitmq::Config/File[rabbitmq-env.config]/ensure: defined content as '{md5}6513cd3ef9805aa61b060ae020df4fd5'
Error: Could not start Service[rabbitmq-server]: Execution of '/sbin/service rabbitmq-server start' returned 1: Starting rabbitmq-server
: FAILED - check /var/log/rabbitmq/startup_{log, _err}
rabbitmq-server.
Wrapped exception:
Execution of '/sbin/service rabbitmq-server start' returned 1: Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/startup_{log, _err}
rabbitmq-server.
Error: /Stage[main]/Rabbitmq::Service/Service[rabbitmq-server]/ensure: change from stopped to running failed: Could not start Service[ra
bbitmq-server]: Execution of '/sbin/service rabbitmq-server start' returned 1: Starting rabbitmq-server: FAILED - check /var/log/rabbitmq/star
tup_{log, _err}
rabbitmq-server.
Notice: /Stage[main]/Rabbitmq::Service/Service[rabbitmq-server]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Rabbitmq/Anchor[rabbitmq::end]: Dependency Service[rabbitmq-server] has failures: true
Warning: /Stage[main]/Rabbitmq/Anchor[rabbitmq::end]: Skipping because of failed dependencies
Notice: /Stage[main]/Main/Packstack::Firewall[amqp_172.16.0.19]/Firewall[001 amqp incoming amqp_172.16.0.19]/ensure: created
Notice: Finished catalog run in 230.28 seconds
[root@oshost ~]#

There are two more log/err files refered to by this log.
One is blank and the other has only one line.
Here they are:
[root@oshost ~]# more /var/log/rabbitmq/startup_err
[root@oshost ~]# more /var/log/rabbitmq/startup_log
ERROR: epmd error for host "oshost": timeout (timed out)
[FAILED]
[root@oshost ~]#

This should also be in the Workarounds page
http://openstack.redhat.com/Workarounds
as other colleagues have encountered this.

Comment 2 Bill Bruns 2014-05-19 00:37:10 UTC
There is a workaround, following the notes by "Rob" on the web page
http://codingiscoding.wordpress.com/2011/04/17/rabbitmq-on-os-x/

The workaround is to insure that the name of the install machine is in the /etc/hosts file with the correct ip address. Then reinitiate the install either by
packstack --allinone
or by
packstack --answer-file=<answer file created by the failed installation, in your home directory>

This workaround should be listed on the Red Hat Workaround page for OpenStack:
http://openstack.redhat.com/Workarounds

Giving at least this workaround will help potential customers that try to use Packstack.

Comment 3 Martin Magr 2014-05-19 15:04:00 UTC
This is epmd (Erlang port mapper daemon) issue. I hit this once too. We will have to investigate how we can avoid this issue. Thanks for the report. Moving bug to correct product.

Comment 4 Mike Ruckman 2014-10-02 16:42:37 UTC
Ran into this during the Juno testday [0] on F21.

Logs:
stdout - http://ur1.ca/iadvc
openstack-setup - http://ur1.ca/iadvf
amqp.pp.log - http://ur1.ca/iadvh

[0] https://openstack.redhat.com/RDO_test_day_Juno_milestone_3

Comment 5 Miroslav Suchý 2015-01-26 16:02:57 UTC
see bug 1185921 for solution