Bug 1199519

+++ This bug was initially created as a clone of Bug #1147224 +++

Description of problem: When using Packstack to install RHOS5 AIO over RHEL7 with SSL enabled Horizon and AMQP, rabbitmq service fails to start. 

10.35.160.137_amqp.pp:                            [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.35.160.137_amqp.pp
Error: Could not start Service[rabbitmq-server]: Execution of '/usr/bin/systemctl start rabbitmq-server' returned 1: Job for rabbitmq-server.service failed. See 'systemctl status rabbitmq-server.service' and 'journalctl -xn' for details.

Version-Release number of selected component (if applicable):
RHEL7
openstack-packstack-2014.1.1-0.41.dev1251.el7ost.noarch
openstack-packstack-puppet-2014.1.1-0.41.dev1251.el7ost.noarch
RHOS5 repo from  17-Sep-2014 

How reproducible:
Every time happened on three setups.

Steps to Reproduce:
1. Generate answer file, other than usual setting (attached answer file) set these two:
CONFIG_HORIZON_SSL=y
CONFIG_AMQP_ENABLE_SSL=y

2. Run packstack with answer file
3. Fails to start rabbitmq.service. 
4. Manual restart doesn't help, for debugging disabled firewall / selinux still can't start service. 

Actual results:
Failed to start rabbitmq service, packstack run failed, see attached logs.

See attached rabbit
 {rabbit,failure_during_boot,
                    {case_clause,{error,{already_started,<0.275.0>}}}}}}}}

Not sure this is OK or wrong, but under /etc/rabbitmq/ssl   there are no files. 

Expected results:
Packstack should manage to start rabbtimq service, plus complete SSL based AIO deployment successfully. 

Additional info:
lbezdick (thanks) pointed me to an upstream puppet problem:
https://bugs.launchpad.net/puppet-neutron/+bug/1356083

Also suggested using new amqp.pp from https://review.openstack.org/#/c/99649/ deployment passed amqp step but failed later on, this time on _nova.pp:  
AMQP server on 10.35.160.137:5671 is unreachableAdd 
Nova logs under folder called nova.pp  

Added both original amqp.pp.org and new version amqp.pp.

--- Additional comment from Gilles Dubreuil on 2014-09-28 07:33:59 EDT ---

Hi Tzach,

Could you please provide rabbitmq logs?

Regards,
Gilles

--- Additional comment from Gilles Dubreuil on 2014-09-28 09:18:48 EDT ---

It seems, although the CONFIG_AMQP_SSL_SELF_SIGNED option is present in the answer file, that certificate file is missing.

The default self-signed certificate and key files are supposed to be:
/etc/pki/tls/certs/amqp_selfcert.pem
/etc/pki/tls/private/amqp_selfkey.pem

Could you please verify and provide files content if existing?

--- Additional comment from Tzach Shefi on 2014-09-29 02:25:06 EDT ---

Certificate files were created, attaching them again plus rabbitmq logs.

--- Additional comment from Tzach Shefi on 2014-09-29 02:25:36 EDT ---



--- Additional comment from Tzach Shefi on 2014-09-29 02:26:24 EDT ---



--- Additional comment from Tzach Shefi on 2014-09-29 02:27:40 EDT ---



--- Additional comment from Gilles Dubreuil on 2014-09-29 03:18:46 EDT ---

Hi Tzach,

Thanks for the update.

Could you please confirm following:
After using the new manifest amqp.pp mentioned by Lukas,
Rabbitmq service installs correctly and is up and running?

If that the case then it's a different issue and I believe we need to track it down separately.

Regards

--- Additional comment from Tzach Shefi on 2014-09-29 04:35:48 EDT ---

Hi Gilles, 

Confirm after using new manifest amqp.pp service looks up and running. 

Service is up and running:
[root@cougar08 rabbitmq]# systemctl -t service -a | grep rabbit
rabbitmq-server.service                                                                   loaded active     running       RabbitMQ broker

Port is open and listening:
Rabbitmq ssl port 5671 looks OK

[root@cougar08 rabbitmq]#  netstat -lnp | grep 5671
tcp6       0      0 :::5671                 :::*                    LISTEN      13589/beam.smp
[root@cougar08 rabbitmq]#

firewall rule added:
-A INPUT -s 10.35.160.137/32 -p tcp -m multiport --dports 5671,5672 -m comment --comment "001 amqp incoming amqp_10.35.160.137" -j ACCEPT

Yet I still can't explain how then Nova can't reach rabbitmq
Notice Nova's journalctl -xn error ->

2014-09-28 11:20:20.967 17723 ERROR oslo.messaging._drivers.impl_rabbit [req-0e21fb20-fbc0-47bd-851f-172210e65d63 - - - - -] AMQP server on 10.35.160.137:5671 is unreachable: Socket closed. Trying again in 30 seconds.
Sep 28 11:20:21 cougar08.scl.lab.tlv.redhat.com cinder-backup[17630]: 2014-09-28 11:20:21.066 17630 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.35.160.137:5671 is unreachable: Socket closed. Trying again in 30 seconds.

Do you still recommend following it up as a new bug for Nova?

--- Additional comment from Gilles Dubreuil on 2014-09-29 07:31:27 EDT ---

Yes, it seems the initial issue has been fixed, having Rabbitmq listening on ssl port.

The other openstack services which cannot reach rabbitmq seems to be related to comment#1 mentioning upstream issue, should effectively be followed up separately.

The amqp.pp patch will also make its way to the build.

--- Additional comment from Tzach Shefi on 2014-09-30 03:18:00 EDT ---

Created new Packstack bug for Nova bug described on comment#8 
https://bugzilla.redhat.com/show_bug.cgi?id=1147823