Bug 1227421 - Overcloud deployment failed - "Resource CREATE failed - status code: 1 - Error running /var/lib/heat-config/heat-config-puppet
Summary: Overcloud deployment failed - "Resource CREATE failed - status code: 1 - Err...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Linux
low
urgent
Target Milestone: ga
: Director
Assignee: Giulio Fidente
QA Contact: Ola Pavlenko
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-02 15:51 UTC by Ola Pavlenko
Modified: 2015-08-05 13:52 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Upstream Nova defaults to appending "novalocal" to host names. This causes problems with our tooling. The resulting nova host names don't correspond to the names expected by Puppet. This caused CREATE_FAILED on ControllerNodesPostDeployment in the Heat stack during Overcloud creation. This fix overrides the upstream default and does not set the "novalocal" suffix. Now post-deployment Puppet configuration reports a CREATE_COMPLETE on the Overcloud Heat stack.
Clone Of:
Environment:
Last Closed: 2015-08-05 13:52:25 UTC
Target Upstream Version:


Attachments (Terms of Use)
deploy_new.log (1.13 MB, text/plain)
2015-06-02 15:51 UTC, Ola Pavlenko
no flags Details
compute_messages.log (247.56 KB, text/plain)
2015-06-03 17:00 UTC, Ola Pavlenko
no flags Details
control_messages.log (256.20 KB, text/plain)
2015-06-03 17:00 UTC, Ola Pavlenko
no flags Details
deploy.log (783.47 KB, text/plain)
2015-06-03 17:01 UTC, Ola Pavlenko
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Ola Pavlenko 2015-06-02 15:51:48 UTC
Created attachment 1033860 [details]
deploy_new.log

Description of problem:
prepared a plan for deployment using unified cli (openstack client)
tried to deploy the overcloud using the openstack overcloud deploy --plan-uuid $UUID command and got the following error:
Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1"" 

Version-Release number of selected component (if applicable):
$ rpm -qa | grep rdo 
rdo-release-kilo-1.noarch
rubygem-rdoc-4.0.0-24.el7.noarch
python-rdomanager-oscplugin-0.0.7-dev45.el7.centos.noarch
[stack@instack ~]$ rpm -qa | grep openstack| grep heat
openstack-heat-engine-2015.1.1-dev11.el7.centos.noarch
openstack-heat-api-cfn-2015.1.1-dev11.el7.centos.noarch
openstack-tripleo-heat-templates-0.8.6-dev15.el7.centos.noarch
openstack-heat-api-2015.1.1-dev11.el7.centos.noarch
openstack-heat-common-2015.1.1-dev11.el7.centos.noarch
openstack-heat-templates-0.0.0-post350.el7.centos.noarch
openstack-heat-api-cloudwatch-2015.1.1-dev11.el7.centos.noarch


How reproducible:
100%

Steps to Reproduce:
1.prepare a plan using unified cli
2.deploy a plan using unified cli

Actual results:
deployment failed

Expected results:
overcloud was successfully deployed

Additional info:

please see attached deploy_new.log with all details.

Comment 3 Ola Pavlenko 2015-06-03 16:59:51 UTC
I've reinstalled my env with new puddle (from June 2nd) and it reproduces.

env: root@vega16.qa.lab.tlv.redhat.com
instack VM: 192.168.122.236

(Leaving the env as is until tomorrow in case anyone would like to take a look)
Versions:

$ rpm -qa | grep rhos
rhos-release-0.62-1.noarch

$ rpm -qa | grep triple
openstack-tripleo-0.0.6-dev1717.el7.centos.noarch
tripleo-common-0.0.1-dev5.el7.centos.noarch
openstack-tripleo-heat-templates-0.8.6-dev15.el7.centos.noarch
openstack-tripleo-image-elements-0.9.6-dev4.el7.centos.noarch
openstack-tripleo-puppet-elements-0.0.1.dev55-2.el7ost.noarch

$ rpm -qa | grep rdo
rdo-release-kilo-1.noarch
rubygem-rdoc-4.0.0-24.el7.noarch
python-rdomanager-oscplugin-0.0.7-dev45.el7.centos.noarch

$ rpm -qa | grep puppet
openstack-puppet-modules-2015.1.2-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1.dev55-2.el7ost.noarch
puppet-3.6.2-3.el7.noarch

$ rpm -qa | grep heat
openstack-heat-engine-2015.1.1-dev11.el7.centos.noarch
openstack-heat-api-cfn-2015.1.1-dev11.el7.centos.noarch
openstack-tripleo-heat-templates-0.8.6-dev15.el7.centos.noarch
openstack-heat-api-2015.1.1-dev11.el7.centos.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-common-2015.1.1-dev11.el7.centos.noarch
openstack-heat-templates-0.0.0-post350.el7.centos.noarch
python-heatclient-0.5.0-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.1-dev11.el7.centos.noarch


[stack@instack ~]$ openstack overcloud deploy --plan-uuid aa1102b1-899a-47ef-9b2e-c2db0e6b2067 --debug &> deploy.log
please see attached deploy.log  for the output of this command.

[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| ffc5245e-96ce-4464-a763-fd215645ad11 | overcloud  | CREATE_FAILED | 2015-06-03T16:14:15Z |
+--------------------------------------+------------+---------------+----------------------+
[stack@instack ~]$ heat resource-list overcloud
+-----------------------------------+--------------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| resource_name                     | physical_resource_id                             | resource_type                                     | resource_status | updated_time         |
+-----------------------------------+--------------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| BlockStorageAllNodesDeployment    | ab1b8e85-917b-4097-8189-2ee34e227ced             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| BlockStorageNodesPostDeployment   |                                                  | OS::TripleO::BlockStoragePostDeployment           | INIT_COMPLETE   | 2015-06-03T16:14:15Z |
| Ceph-Storage                      | 95eb67c0-b7d4-4de6-aa14-7de3249a42a6             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| CephClusterConfig                 | 89d68382-61e1-4fcb-b830-acd8203845d2             | OS::TripleO::CephClusterConfig::SoftwareConfig    | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| CephStorageAllNodesDeployment     | f8214e8e-db4a-45c1-8349-e8ad561de8b1             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| CephStorageCephDeployment         | 4c07364c-556f-4338-8f0e-60b8f4316874             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| CephStorageNodesPostDeployment    |                                                  | OS::TripleO::CephStoragePostDeployment            | INIT_COMPLETE   | 2015-06-03T16:14:15Z |
| Cinder-Storage                    | f33e3e2d-333c-4510-8d08-37f8dbd49794             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| Compute                           | 6d4e7117-75a7-4386-ac52-cd96f70da85b             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ComputeAllNodesDeployment         | 77f50677-c973-423c-b98b-31d5eeebb671             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ComputeCephDeployment             | 479454ea-f663-4892-809f-4da9d6305a5a             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ComputeNodesPostDeployment        | d82cc4d7-998a-4bb3-bac9-2ae145f29bbc             | OS::TripleO::ComputePostDeployment                | CREATE_FAILED   | 2015-06-03T16:14:15Z |
| ControlVirtualIP                  | 0601e644-eea2-4158-af31-5f78a8e5c244             | OS::Neutron::Port                                 | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| Controller                        | d8f637b8-6c11-4f6b-8603-351ea4690d75             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerAllNodesDeployment      | 75ff4376-16ba-4f2a-b28a-1239cd79c388             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerBootstrapNodeConfig     | 3113766c-5898-4bc5-b4ae-ef13b777af54             | OS::TripleO::BootstrapNode::SoftwareConfig        | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerBootstrapNodeDeployment | edf8b092-8173-4260-b8f5-935aba56847b             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerCephDeployment          | bd0c1550-b60f-4c1f-b58f-0c7fc9cc49f8             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerClusterConfig           | 7890ba7c-30f3-46a4-bbbf-07ac1cea1212             | OS::Heat::StructuredConfig                        | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerClusterDeployment       | dc629bed-d0a5-483b-8980-d94c62af1f61             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ControllerNodesPostDeployment     | b179f9ff-fc30-41ab-aca4-6347478e2d86             | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2015-06-03T16:14:15Z |
| ControllerSwiftDeployment         | 1622999d-0903-41ba-885b-40328b4983ef             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| HeatAuthEncryptionKey             | overcloud-HeatAuthEncryptionKey-coede6t72kyt     | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| HorizonSecret                     | overcloud-HorizonSecret-45ixallq4ujg             | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| MysqlClusterUniquePart            | overcloud-MysqlClusterUniquePart-fefkvsaggdzt    | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| MysqlClustercheckPassword         | overcloud-MysqlClustercheckPassword-kntgyt7xmgph | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| MysqlRootPassword                 | overcloud-MysqlRootPassword-qn22slimc6dv         | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ObjectStorageAllNodesDeployment   | 451a174c-a65a-4af0-84da-8a193ad0ee46             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ObjectStorageNodesPostDeployment  | dd0c999e-4c2a-4695-b8a0-57aef5a82cad             | OS::TripleO::ObjectStoragePostDeployment          | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| ObjectStorageSwiftDeployment      | 9a6b2a7a-48bf-4ecb-ac9d-b7451a672cd9             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| PcsdPassword                      | overcloud-PcsdPassword-nqwjff5crppb              | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| PublicVirtualIP                   | d522b729-802e-4160-a9bf-40aa27739ff7             | OS::Neutron::Port                                 | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| RabbitCookie                      | overcloud-RabbitCookie-am6i4h5krmf4              | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| Swift-Storage                     | fb9ba73d-9926-46a7-a99e-92227d7de346             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| SwiftDevicesAndProxyConfig        | 21a41c14-2aeb-4ebb-a448-94978faff0f7             | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
| allNodesConfig                    | 948a52dc-8a84-4543-8f87-20aafbc5895f             | OS::TripleO::AllNodes::SoftwareConfig             | CREATE_COMPLETE | 2015-06-03T16:14:15Z |
+-----------------------------------+--------------------------------------------------+---------------------------------------------------+-----------------+----------------------+
[stack@instack ~]$ heat resource-show overcloud  ComputeNodesPostDeployment
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                                                                                                                                                   |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                                                                                                                                                                      |
| description            |                                                                                                                                                                                                                                                                                                         |
| links                  | http://192.0.2.1:8004/v1/90d8558659544443bb48882548428f73/stacks/overcloud/ffc5245e-96ce-4464-a763-fd215645ad11/resources/ComputeNodesPostDeployment (self)                                                                                                                                             |
|                        | http://192.0.2.1:8004/v1/90d8558659544443bb48882548428f73/stacks/overcloud/ffc5245e-96ce-4464-a763-fd215645ad11 (stack)                                                                                                                                                                                 |
|                        | http://192.0.2.1:8004/v1/90d8558659544443bb48882548428f73/stacks/overcloud-ComputeNodesPostDeployment-puxhkne3t7lu/d82cc4d7-998a-4bb3-bac9-2ae145f29bbc (nested)                                                                                                                                        |
| logical_resource_id    | ComputeNodesPostDeployment                                                                                                                                                                                                                                                                              |
| physical_resource_id   | d82cc4d7-998a-4bb3-bac9-2ae145f29bbc                                                                                                                                                                                                                                                                    |
| required_by            |                                                                                                                                                                                                                                                                                                         |
| resource_name          | ComputeNodesPostDeployment                                                                                                                                                                                                                                                                              |
| resource_status        | CREATE_FAILED                                                                                                                                                                                                                                                                                           |
| resource_status_reason | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1"" |
| resource_type          | OS::TripleO::ComputePostDeployment                                                                                                                                                                                                                                                                      |
| updated_time           | 2015-06-03T16:14:15Z                                                                                                                                                                                                                                                                                    |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@instack ~]$ nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                                                  | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| 45259403-2791-4f0e-98e5-4dafafbd90bf | ov-mwlmrezfff-0-bjbogr7d2lzk-NovaCompute-fqwa6wunhveh | ACTIVE | -          | Running     | ctlplane=192.0.2.13 |
| fa229586-5f7e-4c6a-a5fa-202bf89142ff | ov-ofjbpdc6ygf-0-jqb2csqzdo2z-Controller-zjkisvgzhmos | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ ssh heat-admin@192.0.2.13
The authenticity of host '192.0.2.13 (192.0.2.13)' can't be established.
ECDSA key fingerprint is 16:e2:ce:a9:59:5a:0f:68:a2:5a:16:74:0d:f1:ca:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.13' (ECDSA) to the list of known hosts.
[heat-admin@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]$ sudo -i 
[root@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]# less  /var/log/messages 
[root@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]# date
Wed Jun  3 12:23:11 EDT 2015
[root@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]# less  /var/log/messages 
[root@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]# grep ERROR  /var/log/messages 
Jun  3 12:17:13 localhost os-collect-config: [2015-06-03 12:17:13,899] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/88dc8a53-1c08-4fb2-a44d-1e4b42f27674.pp. [1]
[root@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]# exit
logout
[heat-admin@ov-mwlmrezfff-0-bjbogr7d2lzk-novacompute-fqwa6wunhveh ~]$ exit
logout
Connection to 192.0.2.13 closed.
[stack@instack ~]$ heat stack-delete overcloud
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| ffc5245e-96ce-4464-a763-fd215645ad11 | overcloud  | DELETE_IN_PROGRESS | 2015-06-03T16:14:15Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id                                   | stack_name | stack_status       | creation_time        |
+--------------------------------------+------------+--------------------+----------------------+
| ffc5245e-96ce-4464-a763-fd215645ad11 | overcloud  | DELETE_IN_PROGRESS | 2015-06-03T16:14:15Z |
+--------------------------------------+------------+--------------------+----------------------+
[stack@instack ~]$ sudo vi /etc/sysconfig/iptables
[stack@instack ~]$ heat stack-list
+----+------------+--------------+---------------+
| id | stack_name | stack_status | creation_time |
+----+------------+--------------+---------------+
+----+------------+--------------+---------------+
[stack@instack ~]$ openstack overcloud deploy --plan-uuid aa1102b1-899a-47ef-9b2e-c2db0e6b2067 --debug &> deploy.log
[stack@instack ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| a7ec7ffb-18dd-4c9d-8152-2a482add31ad | overcloud  | CREATE_FAILED | 2015-06-03T16:26:55Z |
+--------------------------------------+------------+---------------+----------------------+
[stack@instack ~]$ heat resource-list overcloud
+-----------------------------------+--------------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| resource_name                     | physical_resource_id                             | resource_type                                     | resource_status | updated_time         |
+-----------------------------------+--------------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| BlockStorageAllNodesDeployment    | 49023b2d-19f9-44ed-8d74-8a3c9af61882             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| BlockStorageNodesPostDeployment   |                                                  | OS::TripleO::BlockStoragePostDeployment           | INIT_COMPLETE   | 2015-06-03T16:26:55Z |
| Ceph-Storage                      | 5713ee9e-72cb-4f83-bcd1-132c27caf505             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| CephClusterConfig                 | f06e0133-7636-488f-a85e-30bd980cd722             | OS::TripleO::CephClusterConfig::SoftwareConfig    | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| CephStorageAllNodesDeployment     | e720698f-7b85-4663-9e68-d413d3310e2b             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| CephStorageCephDeployment         | 43ec507e-8c97-4de1-966e-f8a00318c273             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| CephStorageNodesPostDeployment    |                                                  | OS::TripleO::CephStoragePostDeployment            | INIT_COMPLETE   | 2015-06-03T16:26:55Z |
| Cinder-Storage                    | e2edf784-88ca-45c5-8a5a-48d486bdd2c4             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| Compute                           | 3b965d6e-4746-4381-827b-ca55aad16776             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ComputeAllNodesDeployment         | 845ffa9b-f525-4cb2-a791-cdaab41a5aec             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ComputeCephDeployment             | 6437085f-ffb5-49ca-a369-2ec57e968f4a             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ComputeNodesPostDeployment        | e8afb777-c8e6-44bc-864e-8e5f2d2b36f2             | OS::TripleO::ComputePostDeployment                | CREATE_FAILED   | 2015-06-03T16:26:55Z |
| ControlVirtualIP                  | 8593a9d7-a876-42e7-b14e-0c5b6353f2d9             | OS::Neutron::Port                                 | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| Controller                        | 2daf6c93-690b-4716-b663-9c30d65a8422             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerAllNodesDeployment      | ead221c0-7c8b-43ab-8c14-a9575e122c5a             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerBootstrapNodeConfig     | 9978ee63-001f-4bce-a51c-004413b7e4b3             | OS::TripleO::BootstrapNode::SoftwareConfig        | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerBootstrapNodeDeployment | 4b304d45-5be8-4eef-a577-0fe7256f45b3             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerCephDeployment          | fd2db166-03fe-4567-b9e1-1d3044ebc426             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerClusterConfig           | 04dfb51e-c370-45a9-b1cc-63a5b17a793d             | OS::Heat::StructuredConfig                        | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerClusterDeployment       | d68c5956-0d1f-46ca-bea3-f0b535d63f5a             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ControllerNodesPostDeployment     | 3b94204d-98d2-40a6-be8e-88ae4026389f             | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2015-06-03T16:26:55Z |
| ControllerSwiftDeployment         | fd693945-0594-4cf5-ba89-cf5464c07419             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| HeatAuthEncryptionKey             | overcloud-HeatAuthEncryptionKey-tyg36x7s2ald     | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| HorizonSecret                     | overcloud-HorizonSecret-4mkspk35fnld             | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| MysqlClusterUniquePart            | overcloud-MysqlClusterUniquePart-3eikkgixv7hz    | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| MysqlClustercheckPassword         | overcloud-MysqlClustercheckPassword-v6sbdodmm5zx | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| MysqlRootPassword                 | overcloud-MysqlRootPassword-hmsoymzpkihg         | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ObjectStorageAllNodesDeployment   | 13f5677f-6459-4d49-b3cf-23e21a1c45c6             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ObjectStorageNodesPostDeployment  | bb0dc7c7-74fb-4833-a157-bb735e5b2ab7             | OS::TripleO::ObjectStoragePostDeployment          | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| ObjectStorageSwiftDeployment      | 7b30028b-08db-4e9f-90f0-1404f2c9f09f             | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| PcsdPassword                      | overcloud-PcsdPassword-ujnxiwp53c5n              | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| PublicVirtualIP                   | b1564768-1550-4d02-bebc-96f6fe9c009f             | OS::Neutron::Port                                 | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| RabbitCookie                      | overcloud-RabbitCookie-zzamuibhrz6l              | OS::Heat::RandomString                            | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| Swift-Storage                     | 2eb72681-5c1d-486b-9344-d6573256441e             | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| SwiftDevicesAndProxyConfig        | c96326e8-34f3-4cbc-aca2-fa98356e2098             | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
| allNodesConfig                    | f48c0f10-55b0-4da4-ae7b-67bef067e8a3             | OS::TripleO::AllNodes::SoftwareConfig             | CREATE_COMPLETE | 2015-06-03T16:26:55Z |
+-----------------------------------+--------------------------------------------------+---------------------------------------------------+-----------------+----------------------+
[stack@instack ~]$ heat resource-show overcloud  ComputeNodesPostDeployment
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                                                                                                                                                   |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                                                                                                                                                                      |
| description            |                                                                                                                                                                                                                                                                                                         |
| links                  | http://192.0.2.1:8004/v1/90d8558659544443bb48882548428f73/stacks/overcloud/a7ec7ffb-18dd-4c9d-8152-2a482add31ad/resources/ComputeNodesPostDeployment (self)                                                                                                                                             |
|                        | http://192.0.2.1:8004/v1/90d8558659544443bb48882548428f73/stacks/overcloud/a7ec7ffb-18dd-4c9d-8152-2a482add31ad (stack)                                                                                                                                                                                 |
|                        | http://192.0.2.1:8004/v1/90d8558659544443bb48882548428f73/stacks/overcloud-ComputeNodesPostDeployment-m54udzg5bj3y/e8afb777-c8e6-44bc-864e-8e5f2d2b36f2 (nested)                                                                                                                                        |
| logical_resource_id    | ComputeNodesPostDeployment                                                                                                                                                                                                                                                                              |
| physical_resource_id   | e8afb777-c8e6-44bc-864e-8e5f2d2b36f2                                                                                                                                                                                                                                                                    |
| required_by            |                                                                                                                                                                                                                                                                                                         |
| resource_name          | ComputeNodesPostDeployment                                                                                                                                                                                                                                                                              |
| resource_status        | CREATE_FAILED                                                                                                                                                                                                                                                                                           |
| resource_status_reason | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "Resource CREATE failed: Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1"" |
| resource_type          | OS::TripleO::ComputePostDeployment                                                                                                                                                                                                                                                                      |
| updated_time           | 2015-06-03T16:26:55Z                                                                                                                                                                                                                                                                                    |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@instack ~]$ nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                                                  | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| 6e90957a-e677-459b-bb7a-a7cdc93d6c27 | ov-7djrh43hma3-0-eqypolxayezy-Controller-cam4hpvy5ptg | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 3a1ecb67-4d9a-4e7f-af15-5f36479e276e | ov-ypxvgvmpml-0-4vs2gu3jciuw-NovaCompute-jk2u6hpmdxpn | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ ssh heat-admin@192.0.2.16
The authenticity of host '192.0.2.16 (192.0.2.16)' can't be established.
ECDSA key fingerprint is db:26:84:e3:c9:44:07:27:f8:32:73:9d:30:0b:e6:49.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.16' (ECDSA) to the list of known hosts.
[heat-admin@ov-7djrh43hma3-0-eqypolxayezy-controller-cam4hpvy5ptg ~]$ sudo -i 
[root@ov-7djrh43hma3-0-eqypolxayezy-controller-cam4hpvy5ptg ~]# grep ERROR /var/log/messages 
Jun  3 12:30:07 localhost os-collect-config: [2015-06-03 12:30:07,356] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/225756ee-1bd6-42c1-bc5b-a9c6fa6f0db1.pp. [1]

And same for compute node: $ nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                                                  | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| 6e90957a-e677-459b-bb7a-a7cdc93d6c27 | ov-7djrh43hma3-0-eqypolxayezy-Controller-cam4hpvy5ptg | ACTIVE | -          | Running     | ctlplane=192.0.2.16 |
| 3a1ecb67-4d9a-4e7f-af15-5f36479e276e | ov-ypxvgvmpml-0-4vs2gu3jciuw-NovaCompute-jk2u6hpmdxpn | ACTIVE | -          | Running     | ctlplane=192.0.2.17 |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
[stack@instack ~]$ ssh heat-admin@192.0.2.17
The authenticity of host '192.0.2.17 (192.0.2.17)' can't be established.
ECDSA key fingerprint is e4:bf:4a:f4:96:ed:a8:32:d1:ba:46:0e:38:9f:5e:91.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.0.2.17' (ECDSA) to the list of known hosts.
[heat-admin@ov-ypxvgvmpml-0-4vs2gu3jciuw-novacompute-jk2u6hpmdxpn ~]$ sudo -i 
[root@ov-ypxvgvmpml-0-4vs2gu3jciuw-novacompute-jk2u6hpmdxpn ~]# grep ERROR /var/log/messages 
Jun  3 12:29:53 localhost os-collect-config: [2015-06-03 12:29:53,805] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/499be8d5-0410-4f45-9c48-8c9aa397ef66.pp. [1]


/var/log/messages from both nodes are attached

Comment 4 Ola Pavlenko 2015-06-03 17:00:27 UTC
Created attachment 1034411 [details]
compute_messages.log

Comment 5 Ola Pavlenko 2015-06-03 17:00:51 UTC
Created attachment 1034412 [details]
control_messages.log

Comment 6 Ola Pavlenko 2015-06-03 17:01:27 UTC
Created attachment 1034413 [details]
deploy.log

Comment 7 Ola Pavlenko 2015-06-03 17:23:50 UTC
Used https://repos.fedorapeople.org/repos/openstack-m/docs/internal/master/environments/virtual.html
but with openstack client commands (e.g unified cli)

Comment 8 Giulio Fidente 2015-06-03 17:25:17 UTC
from controller logs I see:

  Could not retrieve facts for ov-7djrh43hma3-0-eqypolxayezy-controller-cam4hpvy5ptg.novalocal

which seems correct, .novalocal suffix should not be there because of:

https://review.openstack.org/#/c/178108/
https://review.openstack.org/#/c/178110/
https://review.openstack.org/#/c/178111/

Comment 9 Ola Pavlenko 2015-06-03 18:22:31 UTC
During the installation of the undercloud I'v enabled RDO, epel, and RDO-Manager...
could be the reason?

Comment 10 Giulio Fidente 2015-06-04 09:50:52 UTC
Should be fixed by:https://review.gerrithub.io/#/c/235162

Comment 14 errata-xmlrpc 2015-08-05 13:52:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.