Bug 1397863 - Overcloud Deployment fails due to rabbitmq-plugins list
Summary: Overcloud Deployment fails due to rabbitmq-plugins list
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: RDO
Classification: Community
Component: openstack-puppet-modules
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: trunk
Assignee: Emilien Macchi
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-23 13:25 UTC by Donny Davis
Modified: 2016-11-23 14:40 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-23 14:40:00 UTC

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Donny Davis 2016-11-23 13:25:23 UTC 
Description of problem:
Overcloud fails to deploy with HA controllers (non ha not tested) due to a dependency on /usr/lib/rabbitmq/bin/rabbitmq-plugins list -E -m

This command fails, and returns no output. 

Version-Release number of selected component (if applicable):

Mitaka
Newton

How reproducible:
100%

Steps to Reproduce:
1. Deploy undercloud
2. Deploy overcloud
3. Run puppet apply -e 'include rabbitmq' --debug --verbose --detailed-exitcodes on failed node to reveal command that is failing

Actual results:
Overcloud deploy fails 

| ControllerNodesPostDeployment               | 77540034-fd02-4682-96ef-9633528efea7          | OS::TripleO::ControllerPostDeployment                                           | CREATE_FAILED      | 2016-11-23T04:26:12 | overcloud                                                                                                |
| ControllerServicesBaseDeployment_Step2      | 5554d089-f352-4274-a572-15b6350c054f          | OS::Heat::StructuredDeployments                                                 | CREATE_FAILED      | 2016-11-23T05:31:00 | overcloud-ControllerNodesPostDeployment-crvexxdlqiio                                                     |
| 0                                           | a29475fc-2c65-4abd-a316-6c13ce4707b2          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED      | 2016-11-23T05:58:20 | overcloud-ControllerNodesPostDeployment-crvexxdlqiio-ControllerServicesBaseDeployment_Step2-ng7pkurce7pp |
| 2                                           | 16614478-6575-44b8-9a0a-ccde4748bf70          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED      | 2016-11-23T05:58:20 | overcloud-ControllerNodesPostDeployment-crvexxdlqiio-ControllerServicesBaseDeployment_Step2-ng7pkurce7pp |
| ComputeNodesPostDeployment                  | a120a8c4-38bc-4cd2-9944-8989ebabb5da          | OS::TripleO::ComputePostDeployment                                              | CREATE_FAILED      | 2016-11-23T04:26:11 | overcloud                                                                                                |
| ComputePuppetDeployment                     | 488edbf4-d85b-4588-b7a4-87aa49593901          | OS::Heat::StructuredDeployments                                                 | CREATE_FAILED      | 2016-11-23T05:30:57 | overcloud-ComputeNodesPostDeployment-juxi6oeglyi6                           
Expected results:
Overcloud deploys

Additional info:

This is a regression, I deployed this exact same configuration last month on a production scale. without issue. This is also not an issue with RHOSP, and appears to be linked to upstream puppet modules. 

Here is the debug output from a failed node. 

Debug: Executing '/usr/lib/rabbitmq/bin/rabbitmq-plugins list -E -m'
Debug: Command failed, retrying
Comment 1 Donny Davis 2016-11-23 13:55:52 UTC 
The offending file is here

/etc/puppet/modules/rabbitmq/lib/puppet/provider/rabbitmq_plugin/rabbitmqplugins.rb

This is the offending command puppet is trying to run 

/usr/lib/rabbitmq/bin/rabbitmq-plugins list -E -m
Comment 2 Donny Davis 2016-11-23 14:39:08 UTC 
Ok, this bug can be closed. I will open a new bug, as this is not a puppet issue. This is a rabbitmq-server issue.
Note You need to log in before you can comment on or make changes to this bug.