Bug 1269002

Summary: instack-undercloud: overcloud HA deployment fails - the rabbitmq doesn't run on the controllers.
Product: [Community] RDO Reporter: Alexander Chuzhoy <sasha>
Component: rdo-managerAssignee: Giulio Fidente <gfidente>
Status: CLOSED CURRENTRELEASE QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: high    
Version: unspecifiedCC: hbrock, jtrowbri, mburns, morazi, ohochman, srevivo
Target Milestone: GA   
Target Release: Liberty   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-01 12:14:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/messages from compute none

Description Alexander Chuzhoy 2015-10-05 22:08:17 UTC
instack-undercloud: overcloud HA deployment fails - the rabbitmq doesn't run on the controllers.


Step to reproduce:
Attempt to deploy overcloud with "openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --timeout 90 --ntp-server x.x.x.x"

Result:
The overcloud deployment fails:
[stack@instack ~]$ heat resource-list -n 5 overcloud|grep FAIL
| ComputeNodesPostDeployment                | 46fec2f4-4542-471c-bbba-f3503297af35          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED      | 2015-10-05T19:48:19 | overcloud                                                                              |
| ControllerBootstrapNodeDeployment         | 646505ac-12fc-4c6d-b8a0-82aefcb81117          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-10-05T19:48:19 | overcloud                                                                              |
| ControllerClusterDeployment               | 51ab7384-a669-4d7f-80be-2e706380a2c4          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-10-05T19:48:19 | overcloud                                                                              |
| ControllerSwiftDeployment                 | 7022a48b-7a82-43ca-83b9-322229d27ac2          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-10-05T19:48:19 | overcloud                                                                              |
| VipDeployment                             | 32299bf3-a1b5-4432-8cfc-99cd49446dfe          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-10-05T19:48:19 | overcloud                                                                              |
| ComputePuppetDeployment                   | 9f5d2a90-c8cc-467f-ade1-9d00a93cc1a3          | OS::Heat::StructuredDeployments                   | CREATE_FAILED      | 2015-10-05T20:00:03 | overcloud-ComputeNodesPostDeployment-himyngfw7azo    







[stack@instack ~]$ heat resource-show overcloud  ComputeNodesPostDeployment
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                            |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                               |
| creation_time          | 2015-10-05T19:48:19                                                                                                                                              |
| description            |                                                                                                                                                                  |
| links                  | http://192.0.2.1:8004/v1/b4752199bdb2492faaee42746a11254d/stacks/overcloud/1c9c8374-ce36-47d8-a8de-d1d77690bdc2/resources/ComputeNodesPostDeployment (self)      |
|                        | http://192.0.2.1:8004/v1/b4752199bdb2492faaee42746a11254d/stacks/overcloud/1c9c8374-ce36-47d8-a8de-d1d77690bdc2 (stack)                                          |
|                        | http://192.0.2.1:8004/v1/b4752199bdb2492faaee42746a11254d/stacks/overcloud-ComputeNodesPostDeployment-himyngfw7azo/46fec2f4-4542-471c-bbba-f3503297af35 (nested) |
| logical_resource_id    | ComputeNodesPostDeployment                                                                                                                                       |
| physical_resource_id   | 46fec2f4-4542-471c-bbba-f3503297af35                                                                                                                             |
| required_by            |                                                                                                                                                                  |
| resource_name          | ComputeNodesPostDeployment                                                                                                                                       |
| resource_status        | CREATE_FAILED                                                                                                                                                    |
| resource_status_reason | CREATE aborted                                                                                                                                                   |
| resource_type          | OS::TripleO::ComputePostDeployment                                                                                                                               |
| updated_time           | 2015-10-05T19:48:19                                                                                                                                              |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+


Repeating messages on the compute:
ct  5 22:01:07 localhost neutron-openvswitch-agent: 2015-10-05 22:01:07.948 5684 ERROR oslo.messaging._drivers.impl_rabbit [req-f4b95faa-02c1-4600-9504-80aed60d4f07 - - - - -] AMQP server on 192.0.2.19:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
Oct  5 22:01:07 localhost neutron-openvswitch-agent: 2015-10-05 22:01:07.949 5684 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.0.2.19:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.



the rabbitmq doesn't run on the controllers.

Comment 1 Alexander Chuzhoy 2015-10-05 22:13:05 UTC
Created attachment 1080083 [details]
/var/log/messages from compute

Comment 2 Alexander Chuzhoy 2015-10-05 22:14:00 UTC
Environment:
instack-0.0.8-dev4.el7.centos.noarch
instack-undercloud-2.1.3-dev219.el7.centos.noarch

Comment 3 Alexander Chuzhoy 2015-10-05 22:15:15 UTC
Used Centos 7.1 and followed the procedure here:
http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_deployment_cli.html

Comment 4 Mike Orazi 2015-10-13 21:27:40 UTC
This may be fixed by some combination of:  https://bugzilla.redhat.com/show_bug.cgi?id=1270964 and https://bugzilla.redhat.com/show_bug.cgi?id=1270325

Can we confirm whether the fixes for those 2 issues makes this go away?

Comment 5 Giulio Fidente 2015-10-14 08:16:34 UTC
Alex, I was unable to reproduce this; can you post which version of the openstack-tripleo-heat-templates you have installed on the undercloud and which version of openstack-puppet-modules you have installed on one of the overcloud controllers?

Comment 6 Alexander Chuzhoy 2015-10-14 14:39:26 UTC
I re-deployed the setup (newer version).
I'm unable to reproduce this so far.