Bug 1397863

Summary:	Overcloud Deployment fails due to rabbitmq-plugins list
Product:	[Community] RDO	Reporter:	Donny Davis <dondavis>
Component:	openstack-puppet-modules	Assignee:	Emilien Macchi <emacchi>
Status:	CLOSED NOTABUG	QA Contact:	Shai Revivo <srevivo>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	srevivo
Target Milestone:	---
Target Release:	trunk
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-23 14:40:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Donny Davis 2016-11-23 13:25:23 UTC

Description of problem:
Overcloud fails to deploy with HA controllers (non ha not tested) due to a dependency on /usr/lib/rabbitmq/bin/rabbitmq-plugins list -E -m

This command fails, and returns no output. 

Version-Release number of selected component (if applicable):

Mitaka
Newton

How reproducible:
100%

Steps to Reproduce:
1. Deploy undercloud
2. Deploy overcloud
3. Run puppet apply -e 'include rabbitmq' --debug --verbose --detailed-exitcodes on failed node to reveal command that is failing

Actual results:
Overcloud deploy fails 

| ControllerNodesPostDeployment               | 77540034-fd02-4682-96ef-9633528efea7          | OS::TripleO::ControllerPostDeployment                                           | CREATE_FAILED      | 2016-11-23T04:26:12 | overcloud                                                                                                |
| ControllerServicesBaseDeployment_Step2      | 5554d089-f352-4274-a572-15b6350c054f          | OS::Heat::StructuredDeployments                                                 | CREATE_FAILED      | 2016-11-23T05:31:00 | overcloud-ControllerNodesPostDeployment-crvexxdlqiio                                                     |
| 0                                           | a29475fc-2c65-4abd-a316-6c13ce4707b2          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED      | 2016-11-23T05:58:20 | overcloud-ControllerNodesPostDeployment-crvexxdlqiio-ControllerServicesBaseDeployment_Step2-ng7pkurce7pp |
| 2                                           | 16614478-6575-44b8-9a0a-ccde4748bf70          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED      | 2016-11-23T05:58:20 | overcloud-ControllerNodesPostDeployment-crvexxdlqiio-ControllerServicesBaseDeployment_Step2-ng7pkurce7pp |
| ComputeNodesPostDeployment                  | a120a8c4-38bc-4cd2-9944-8989ebabb5da          | OS::TripleO::ComputePostDeployment                                              | CREATE_FAILED      | 2016-11-23T04:26:11 | overcloud                                                                                                |
| ComputePuppetDeployment                     | 488edbf4-d85b-4588-b7a4-87aa49593901          | OS::Heat::StructuredDeployments                                                 | CREATE_FAILED      | 2016-11-23T05:30:57 | overcloud-ComputeNodesPostDeployment-juxi6oeglyi6                           
Expected results:
Overcloud deploys

Additional info:

This is a regression, I deployed this exact same configuration last month on a production scale. without issue. This is also not an issue with RHOSP, and appears to be linked to upstream puppet modules. 

Here is the debug output from a failed node. 

Debug: Executing '/usr/lib/rabbitmq/bin/rabbitmq-plugins list -E -m'
Debug: Command failed, retrying

Comment 1 Donny Davis 2016-11-23 13:55:52 UTC

The offending file is here

/etc/puppet/modules/rabbitmq/lib/puppet/provider/rabbitmq_plugin/rabbitmqplugins.rb

This is the offending command puppet is trying to run 

/usr/lib/rabbitmq/bin/rabbitmq-plugins list -E -m

Comment 2 Donny Davis 2016-11-23 14:39:08 UTC

Ok, this bug can be closed. I will open a new bug, as this is not a puppet issue. This is a rabbitmq-server issue.