Bug 1147224

Summary: Packstack install AMQP with SSL, fails to start rabbitmq service
Product: Red Hat OpenStack Reporter: Tzach Shefi <tshefi>
Component: openstack-packstackAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED CURRENTRELEASE QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 6)CC: aortega, derekh, gdubreui, ichavero, srevivo, tshefi, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1199519 (view as bug list) Environment:
Last Closed: 2015-08-27 19:05:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1199519    
Attachments:
Description Flags
Logs files
none
Certificate files
none
Selfcert
none
rabbitmq log none

Description Tzach Shefi 2014-09-28 08:28:35 UTC 
Created attachment 941967 [details]
Logs files

Description of problem: When using Packstack to install RHOS5 AIO over RHEL7 with SSL enabled Horizon and AMQP, rabbitmq service fails to start. 

10.35.160.137_amqp.pp:                            [ ERROR ]
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.35.160.137_amqp.pp
Error: Could not start Service[rabbitmq-server]: Execution of '/usr/bin/systemctl start rabbitmq-server' returned 1: Job for rabbitmq-server.service failed. See 'systemctl status rabbitmq-server.service' and 'journalctl -xn' for details.

Version-Release number of selected component (if applicable):
RHEL7
openstack-packstack-2014.1.1-0.41.dev1251.el7ost.noarch
openstack-packstack-puppet-2014.1.1-0.41.dev1251.el7ost.noarch
RHOS5 repo from  17-Sep-2014 

How reproducible:
Every time happened on three setups.

Steps to Reproduce:
1. Generate answer file, other than usual setting (attached answer file) set these two:
CONFIG_HORIZON_SSL=y
CONFIG_AMQP_ENABLE_SSL=y

2. Run packstack with answer file
3. Fails to start rabbitmq.service. 
4. Manual restart doesn't help, for debugging disabled firewall / selinux still can't start service. 

Actual results:
Failed to start rabbitmq service, packstack run failed, see attached logs.

See attached rabbit
 {rabbit,failure_during_boot,
                    {case_clause,{error,{already_started,<0.275.0>}}}}}}}}

Not sure this is OK or wrong, but under /etc/rabbitmq/ssl   there are no files. 

Expected results:
Packstack should manage to start rabbtimq service, plus complete SSL based AIO deployment successfully. 

Additional info:
lbezdick (thanks) pointed me to an upstream puppet problem:
https://bugs.launchpad.net/puppet-neutron/+bug/1356083

Also suggested using new amqp.pp from https://review.openstack.org/#/c/99649/ deployment passed amqp step but failed later on, this time on _nova.pp:  
AMQP server on 10.35.160.137:5671 is unreachableAdd 
Nova logs under folder called nova.pp  

Added both original amqp.pp.org and new version amqp.pp.
Comment 1 Gilles Dubreuil 2014-09-28 11:33:59 UTC 
Hi Tzach,

Could you please provide rabbitmq logs?

Regards,
Gilles
Comment 2 Gilles Dubreuil 2014-09-28 13:18:48 UTC 
It seems, although the CONFIG_AMQP_SSL_SELF_SIGNED option is present in the answer file, that certificate file is missing.

The default self-signed certificate and key files are supposed to be:
/etc/pki/tls/certs/amqp_selfcert.pem
/etc/pki/tls/private/amqp_selfkey.pem

Could you please verify and provide files content if existing?
Comment 3 Tzach Shefi 2014-09-29 06:25:06 UTC 
Certificate files were created, attaching them again plus rabbitmq logs.
Comment 4 Tzach Shefi 2014-09-29 06:25:36 UTC 
Created attachment 942175 [details]
Certificate files
Comment 5 Tzach Shefi 2014-09-29 06:26:24 UTC 
Created attachment 942176 [details]
Selfcert
Comment 6 Tzach Shefi 2014-09-29 06:27:40 UTC 
Created attachment 942177 [details]
rabbitmq log
Comment 7 Gilles Dubreuil 2014-09-29 07:18:46 UTC 
Hi Tzach,

Thanks for the update.

Could you please confirm following:
After using the new manifest amqp.pp mentioned by Lukas,
Rabbitmq service installs correctly and is up and running?

If that the case then it's a different issue and I believe we need to track it down separately.

Regards
Comment 8 Tzach Shefi 2014-09-29 08:35:48 UTC 
Hi Gilles, 

Confirm after using new manifest amqp.pp service looks up and running. 

Service is up and running:
[root@cougar08 rabbitmq]# systemctl -t service -a | grep rabbit
rabbitmq-server.service                                                                   loaded active     running       RabbitMQ broker

Port is open and listening:
Rabbitmq ssl port 5671 looks OK

[root@cougar08 rabbitmq]#  netstat -lnp | grep 5671
tcp6       0      0 :::5671                 :::*                    LISTEN      13589/beam.smp
[root@cougar08 rabbitmq]#

firewall rule added:
-A INPUT -s 10.35.160.137/32 -p tcp -m multiport --dports 5671,5672 -m comment --comment "001 amqp incoming amqp_10.35.160.137" -j ACCEPT

Yet I still can't explain how then Nova can't reach rabbitmq
Notice Nova's journalctl -xn error ->

2014-09-28 11:20:20.967 17723 ERROR oslo.messaging._drivers.impl_rabbit [req-0e21fb20-fbc0-47bd-851f-172210e65d63 - - - - -] AMQP server on 10.35.160.137:5671 is unreachable: Socket closed. Trying again in 30 seconds.
Sep 28 11:20:21 cougar08.scl.lab.tlv.redhat.com cinder-backup[17630]: 2014-09-28 11:20:21.066 17630 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.35.160.137:5671 is unreachable: Socket closed. Trying again in 30 seconds.

Do you still recommend following it up as a new bug for Nova?
Comment 9 Gilles Dubreuil 2014-09-29 11:31:27 UTC 
Yes, it seems the initial issue has been fixed, having Rabbitmq listening on ssl port.

The other openstack services which cannot reach rabbitmq seems to be related to comment#1 mentioning upstream issue, should effectively be followed up separately.

The amqp.pp patch will also make its way to the build.
Comment 10 Tzach Shefi 2014-09-30 07:18:00 UTC 
Created new Packstack bug for Nova bug described on comment#8 
https://bugzilla.redhat.com/show_bug.cgi?id=1147823
Comment 12 Ivan Chavero 2015-08-27 19:05:04 UTC 
current icehouse packstack code does not have this problem.