Bug 1313514 - rhosp 8 beta 7 deployment failed
rhosp 8 beta 7 deployment failed
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
urgent Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Angus Thomas
Arik Chernetsky
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-01 13:21 EST by bigswitch
Modified: 2016-04-26 19:13 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-10 07:18:28 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
soreport1 (7.57 MB, application/x-xz)
2016-03-01 13:21 EST, bigswitch
no flags Details
sosreport-compute (6.41 MB, application/x-xz)
2016-03-01 13:23 EST, bigswitch
no flags Details

  None (edit)
Description bigswitch 2016-03-01 13:21:31 EST
Created attachment 1132005 [details]
soreport1

Description of problem:
Attempt to deploy rhosp 8 beta 7 with vlan and linux bonding failed. This is one controller and one compute node deployment.
sosreport from both controller and compute is uploaded to box with bugzilla number.

[stack@c5220-01 ~]$ openstack overcloud deploy -e /home/stack/templates/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 1 --ceph-storage-scale 0 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --timeout 60 --debug 2>&1 | tee haha.log



curl -g -i -X GET -H 'X-Auth-Token: {SHA1}a7e8163e7393944ce930c93ae29be13338c91be9' -H 'Content-Type: application/json' -H 'X-Auth-Url: http://192.168.0.1:5000/v2.0' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud-ComputeCephDeployment-qgaxvlkdk54n/1dae7af0-045f-4fda-8e90-262b1e435435/resources

HTTP/1.1 200 OK
date: Tue, 01 Mar 2016 17:59:56 GMT
connection: keep-alive
content-type: application/json; charset=UTF-8
content-length: 783
x-openstack-request-id: req-a6f08769-57a6-4492-89af-9ebbeb996fc3

{"resources": [{"parent_resource": "ComputeCephDeployment", "resource_name": "0", "links": [{"href": "http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud-ComputeCephDeployment-qgaxvlkdk54n/1dae7af0-045f-4fda-8e90-262b1e435435/resources/0", "rel": "self"}, {"href": "http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud-ComputeCephDeployment-qgaxvlkdk54n/1dae7af0-045f-4fda-8e90-262b1e435435", "rel": "stack"}], "logical_resource_id": "0", "creation_time": "2016-03-01T17:54:08", "resource_status": "CREATE_COMPLETE", "updated_time": "2016-03-01T17:54:08", "required_by": [], "resource_status_reason": "state changed", "physical_resource_id": "64b5db43-5d9e-4e27-8e8b-83d0467a52f3", "resource_type": "OS::Heat::StructuredDeployment"}]}

Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 991, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 342, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 274, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.
clean_up DeployOvercloud: Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 112, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 255, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 991, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 342, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 274, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.

END return value: 1
  Stack CREATE started
2016-03-01 17:58:53 [overcloud-ControllerNodesPostDeployment-djf2fqndaesc-ControllerOvercloudServicesDeployment_Step4-zetcvrnpk4kz]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-03-01 17:58:54 [NetworkDeployment]: SIGNAL_COMPLETE  Unknown
2016-03-01 17:58:54 [overcloud-ControllerNodesPostDeployment-djf2fqndaesc-ControllerOvercloudServicesDeployment_Step4-zetcvrnpk4kz]: UPDATE_IN_PROGRESS  Stack UPDATE started
2016-03-01 17:58:54 [0]: SIGNAL_COMPLETE  Unknown
2016-03-01 17:58:54 [0]: SIGNAL_COMPLETE  Unknown
2016-03-01 17:58:55 [0]: CREATE_IN_PROGRESS  state changed
2016-03-01 17:58:55 [0]: SIGNAL_COMPLETE  Unknown
2016-03-01 17:58:55 [0]: SIGNAL_COMPLETE  Unknown
2016-03-01 17:59:38 [ControllerNodesPostDeployment]: CREATE_FAILED  CREATE aborted
2016-03-01 17:59:38 [ComputeNodesPostDeployment]: CREATE_FAILED  CREATE aborted
2016-03-01 17:59:38 [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.ComputeAllNodesValidationDeployment: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
Stack overcloud CREATE_FAILED
[stack@c5220-01 ~]$ heat resource-list overcloud
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time        |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+
| AllNodesValidationConfig                  | f3a2a342-88a3-487c-af62-82e617813545          | OS::TripleO::AllNodes::Validation                 | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| BlockStorageAllNodesDeployment            | b00dc4de-017b-4be4-9644-6d17e3f30427          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| BlockStorageAllNodesValidationDeployment  | ec71edee-8374-47d7-b9e6-f01142106c90          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| BlockStorageNodesPostDeployment           |                                               | OS::TripleO::BlockStoragePostDeployment           | INIT_COMPLETE   | 2016-03-01T17:43:54 |
| CephStorageAllNodesDeployment             | 46052174-7deb-4dc0-b1ad-c1ebb7469667          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| CephStorageAllNodesValidationDeployment   | 4c45f008-1259-41e4-b317-e26d7e5aa634          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| CephStorageCephDeployment                 | e79c2aec-91ad-417a-8d99-36ec563b41fc          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| CephStorageNodesPostDeployment            |                                               | OS::TripleO::CephStoragePostDeployment            | INIT_COMPLETE   | 2016-03-01T17:43:54 |
| ComputeAllNodesValidationDeployment       | 43cf7036-2b47-4680-b243-9ec7091976cf          | OS::Heat::StructuredDeployments                   | CREATE_FAILED   | 2016-03-01T17:43:54 |
| ComputeNodesPostDeployment                | 091876a5-1db6-4edf-bbcb-e36f3262ff97          | OS::TripleO::ComputePostDeployment                | CREATE_FAILED   | 2016-03-01T17:43:54 |
| ControllerAllNodesDeployment              | cf1bc3d3-7c3e-40e8-bb22-a69ecde3a4d3          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerAllNodesValidationDeployment    | 2273575d-256d-4dda-974f-bbe6510c8d52          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerBootstrapNodeConfig             | e81aad47-75a6-4d8b-9a54-415bf1f7ace5          | OS::TripleO::BootstrapNode::SoftwareConfig        | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerBootstrapNodeDeployment         | 57ea99a1-0864-424d-a037-6e56397d85b5          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerCephDeployment                  | 466ecc00-0bad-4366-b9af-72f91e7867b8          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerClusterConfig                   | 7ac0f2de-79b8-4fe1-ad69-62942901dd3c          | OS::Heat::StructuredConfig                        | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerClusterDeployment               | 60f1d9ea-81ba-4162-838e-f194a8e836e5          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ControllerNodesPostDeployment             | 71fb57e6-b759-419f-8e31-5931d25426c3          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2016-03-01T17:43:54 |
| ControllerSwiftDeployment                 | cf25d67f-b1f6-457c-a85c-299302db83d6          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| ObjectStorageAllNodesValidationDeployment | 392b03c9-d7a7-4d01-8886-921ab8da2c6c          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| VipDeployment                             | e3527b5f-270c-47b3-9cab-656ec588f72a          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:54 |
| AllNodesExtraConfig                       | 68cd843c-cbf1-4cad-b8f4-3b55e72b6c7b          | OS::TripleO::AllNodesExtraConfig                  | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| BlockStorage                              | 865c65cd-90da-4631-8e48-1ee9f71d7072          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| CephClusterConfig                         | c5c9b3e5-39ad-4479-8d3d-aef58953b33e          | OS::TripleO::CephClusterConfig::SoftwareConfig    | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| CephStorage                               | 625bcc2c-8e43-414e-a3d1-24459baba1b6          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| Compute                                   | 39d7c08c-d565-4123-a26a-fe7ae4dbd77e          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ComputeAllNodesDeployment                 | 50b271d9-b985-4f43-92e4-2a74cf7b3d39          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ComputeCephDeployment                     | 1dae7af0-045f-4fda-8e90-262b1e435435          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ControlVirtualIP                          | 8e2955ed-4aba-40ad-b208-dbc94bd52bef          | OS::Neutron::Port                                 | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| Controller                                | c9b8b08b-438d-46d5-906f-3a436131f375          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ControllerIpListMap                       | 0bc9bffa-0cc1-4338-86f7-743fd38a8704          | OS::TripleO::Network::Ports::NetIpListMap         | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| EndpointMap                               | 1fabbe73-dbcf-4b1e-a38c-5584d93b6542          | OS::TripleO::EndpointMap                          | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| HeatAuthEncryptionKey                     | overcloud-HeatAuthEncryptionKey-c3k5x25wkbuu  | OS::Heat::RandomString                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| HorizonSecret                             | overcloud-HorizonSecret-k5v55lqzalma          | OS::Heat::RandomString                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| InternalApiVirtualIP                      | 8055918b-c505-43ef-a885-45f923aea629          | OS::TripleO::Network::Ports::InternalApiVipPort   | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| MysqlClusterUniquePart                    | overcloud-MysqlClusterUniquePart-xreblerxe5qj | OS::Heat::RandomString                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| MysqlRootPassword                         | overcloud-MysqlRootPassword-rqkabdqvg32n      | OS::Heat::RandomString                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| Networks                                  | 1c251b74-01b8-4a66-ab03-93f5e8f2ee3d          | OS::TripleO::Network                              | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ObjectStorage                             | 074f4f30-5058-4308-9733-83992d33d37c          | OS::Heat::ResourceGroup                           | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ObjectStorageAllNodesDeployment           | d3f09d9c-c558-4d91-9dca-f1e374da9e3b          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ObjectStorageNodesPostDeployment          | 2079983e-9321-4988-a228-99d42884f467          | OS::TripleO::ObjectStoragePostDeployment          | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| ObjectStorageSwiftDeployment              | 398fa5c0-5bcf-4a2a-9e1b-2d816d0ca986          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| PcsdPassword                              | overcloud-PcsdPassword-dij4kuhw5ss2           | OS::Heat::RandomString                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| PublicVirtualIP                           | 61bcb73c-d4af-4194-a55b-04a008bac5d7          | OS::TripleO::Network::Ports::ExternalVipPort      | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| RabbitCookie                              | overcloud-RabbitCookie-euke3l23hnpx           | OS::Heat::RandomString                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| RedisVirtualIP                            | 8db47c46-6668-47ca-905a-883f96e4706e          | OS::TripleO::Network::Ports::RedisVipPort         | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| StorageMgmtVirtualIP                      | ceddd403-0cfc-4bb1-a0dd-3b78611c2b34          | OS::TripleO::Network::Ports::StorageMgmtVipPort   | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| StorageVirtualIP                          | 7028f098-7c86-4b69-a889-e30acd093ddd          | OS::TripleO::Network::Ports::StorageVipPort       | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| SwiftDevicesAndProxyConfig                | ffa57c89-469a-4d37-80cc-ac3bf6c69151          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| VipConfig                                 | 13d93df6-2659-4585-bae1-7f9dbb38cdf6          | OS::TripleO::VipConfig                            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| VipMap                                    | 33c6a918-1a8f-452b-88b1-a88c14bb0d16          | OS::TripleO::Network::Ports::NetVipMap            | CREATE_COMPLETE | 2016-03-01T17:43:55 |
| allNodesConfig                            | 9c5fc304-56e0-4fa1-8b46-907e97888ff9          | OS::TripleO::AllNodes::SoftwareConfig             | CREATE_COMPLETE | 2016-03-01T17:43:55 |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+
[stack@c5220-01 ~]$ heat resource-show overcloud ControllerNodesPostDeployment
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                 |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                                    |
| creation_time          | 2016-03-01T17:43:54                                                                                                                                                   |
| description            |                                                                                                                                                                       |
| links                  | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud/4fe63c70-796e-4ac4-b3c4-a7d618dee0ed/resources/ControllerNodesPostDeployment (self)      |
|                        | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud/4fe63c70-796e-4ac4-b3c4-a7d618dee0ed (stack)                                             |
|                        | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud-ControllerNodesPostDeployment-djf2fqndaesc/71fb57e6-b759-419f-8e31-5931d25426c3 (nested) |
| logical_resource_id    | ControllerNodesPostDeployment                                                                                                                                         |
| physical_resource_id   | 71fb57e6-b759-419f-8e31-5931d25426c3                                                                                                                                  |
| required_by            | BlockStorageNodesPostDeployment                                                                                                                                       |
|                        | CephStorageNodesPostDeployment                                                                                                                                        |
| resource_name          | ControllerNodesPostDeployment                                                                                                                                         |
| resource_status        | CREATE_FAILED                                                                                                                                                         |
| resource_status_reason | CREATE aborted                                                                                                                                                        |
| resource_type          | OS::TripleO::ControllerPostDeployment                                                                                                                                 |
| updated_time           | 2016-03-01T17:43:54                                                                                                                                                   |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@c5220-01 ~]$ heat resource-show overcloud ComputeNodesPostDeployment
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                              |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                                 |
| creation_time          | 2016-03-01T17:43:54                                                                                                                                                |
| description            |                                                                                                                                                                    |
| links                  | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud/4fe63c70-796e-4ac4-b3c4-a7d618dee0ed/resources/ComputeNodesPostDeployment (self)      |
|                        | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud/4fe63c70-796e-4ac4-b3c4-a7d618dee0ed (stack)                                          |
|                        | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud-ComputeNodesPostDeployment-oc5acsstn36q/091876a5-1db6-4edf-bbcb-e36f3262ff97 (nested) |
| logical_resource_id    | ComputeNodesPostDeployment                                                                                                                                         |
| physical_resource_id   | 091876a5-1db6-4edf-bbcb-e36f3262ff97                                                                                                                               |
| required_by            |                                                                                                                                                                    |
| resource_name          | ComputeNodesPostDeployment                                                                                                                                         |
| resource_status        | CREATE_FAILED                                                                                                                                                      |
| resource_status_reason | CREATE aborted                                                                                                                                                     |
| resource_type          | OS::TripleO::ComputePostDeployment                                                                                                                                 |
| updated_time           | 2016-03-01T17:43:54                                                                                                                                                |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@c5220-01 ~]$ heat resource-show overcloud ComputeAllNodesValidationDeployment
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                       |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                           |
|                        |   "deploy_stderrs": null,                                                                                                                                                   |
|                        |   "deploy_stdouts": null,                                                                                                                                                   |
|                        |   "deploy_status_codes": null                                                                                                                                               |
|                        | }                                                                                                                                                                           |
| creation_time          | 2016-03-01T17:43:54                                                                                                                                                         |
| description            |                                                                                                                                                                             |
| links                  | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud/4fe63c70-796e-4ac4-b3c4-a7d618dee0ed/resources/ComputeAllNodesValidationDeployment (self)      |
|                        | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud/4fe63c70-796e-4ac4-b3c4-a7d618dee0ed (stack)                                                   |
|                        | http://192.168.0.1:8004/v1/f4f5d6685a0c4a398775946b100a1dc9/stacks/overcloud-ComputeAllNodesValidationDeployment-rv3ytglpxia5/43cf7036-2b47-4680-b243-9ec7091976cf (nested) |
| logical_resource_id    | ComputeAllNodesValidationDeployment                                                                                                                                         |
| physical_resource_id   | 43cf7036-2b47-4680-b243-9ec7091976cf                                                                                                                                        |
| required_by            |                                                                                                                                                                             |
| resource_name          | ComputeAllNodesValidationDeployment                                                                                                                                         |
| resource_status        | CREATE_FAILED                                                                                                                                                               |
| resource_status_reason | resources.ComputeAllNodesValidationDeployment: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1        |
| resource_type          | OS::Heat::StructuredDeployments                                                                                                                                             |
| updated_time           | 2016-03-01T17:43:54                                                                                                                                                         |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@c5220-01 ~]$


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 bigswitch 2016-03-01 13:23 EST
Created attachment 1132006 [details]
sosreport-compute
Comment 3 bigswitch 2016-03-01 13:23:37 EST
sosreport is attached
Comment 4 bigswitch 2016-03-01 13:38:32 EST
Added journalctl output from controller:

[root@overcloud-controller-0 tmp]# journalctl -fn 100000 | grep -ni error
1513:Mar 01 12:51:39 localhost.localdomain cloud-init[877]: 2016-03-01 12:51:39,266 - util.py[WARNING]: Route info failed: Unexpected error while running command.
1519:Mar 01 12:51:39 localhost.localdomain cloud-init[877]: [CLOUDINIT] util.py[WARNING]: Route info failed: Unexpected error while running command.
1525:Mar 01 12:51:39 localhost.localdomain cloud-init[877]: [CLOUDINIT] util.py[DEBUG]: Route info failed: Unexpected error while running command.
1538:                                                       ProcessExecutionError: Unexpected error while running command.
2485:Mar 01 12:52:16 overcloud-controller-0.localdomain dracut[3561]: -rwxr-xr-x   1 root     root          145 Jan 15 03:06 usr/bin/kdump-error-handler.sh
2731:Mar 01 12:52:17 overcloud-controller-0.localdomain dracut[3561]: -rw-r--r--   1 root     root          993 Jan 15 03:06 usr/lib/systemd/system/kdump-error-handler.service
2884:Mar 01 12:52:17 overcloud-controller-0.localdomain dracut[3561]: lrwxrwxrwx   1 root     root           22 Mar  1 12:52 usr/lib64/libgpg-error.so -> libgpg-error.so.0.10.0
2885:Mar 01 12:52:17 overcloud-controller-0.localdomain dracut[3561]: lrwxrwxrwx   1 root     root           22 Mar  1 12:52 usr/lib64/libgpg-error.so.0 -> libgpg-error.so.0.10.0
2886:Mar 01 12:52:17 overcloud-controller-0.localdomain dracut[3561]: -rwxr-xr-x   1 root     root        19384 Mar  1 12:52 usr/lib64/libgpg-error.so.0.10.0
5079:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:    error: Resource start-up disabled since no STONITH resources have been defined
5080:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:    error: Either configure some or disable STONITH with the stonith-enabled option
5081:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:    error: NOTE: Clusters with shared data need STONITH to ensure data integrity
5084:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:   notice: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
5089:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:    error: Resource start-up disabled since no STONITH resources have been defined
5090:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:    error: Either configure some or disable STONITH with the stonith-enabled option
5091:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:    error: NOTE: Clusters with shared data need STONITH to ensure data integrity
5094:Mar 01 12:56:58 overcloud-controller-0.localdomain pengine[30046]:   notice: Configuration ERRORs found during PE processing.  Please run "crm_verify -L" to identify issues.
5108:Mar 01 12:57:02 overcloud-controller-0.localdomain os-collect-config[7424]: e]/returns: executed successfully\u001b[0m\n\u001b[mNotice: Pacemaker has reported quorum achieved\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/message: defined 'message' as 'Pacemaker has reported quorum achieved'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Tripleo::Loadbalancer/Haproxy::Listen[redis]/Concat::Fragment[haproxy-redis_listen_block]/File[/var/lib/puppet/concat/_etc_haproxy_haproxy.cfg/fragments/20-redis-00_haproxy-redis_listen_block]/ensure: defined content as '{md5}f11dc73be476a8ce3267d965fedc63f1'\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]: Triggered 'refresh' from 41 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: content changed '{md5}1f337186b0e1ba5ee82760cb437fb810' to '{md5}40706bafaaa45c5cdc110ab5b717f865'\u001b[0m\n\u001b[mNotice: /File[/etc/haproxy/haproxy.cfg]/seluser: seluser changed 'unconfined_u' to 'system_u'\u001b[0m\n\u001b[mNotice: Finished catalog run in 43.66 seconds\u001b[0m\n", "deploy_stderr": "\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user'
5109:Mar 01 12:57:02 overcloud-controller-0.localdomain os-collect-config[7424]: returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0}
5213:Mar 01 12:57:02 overcloud-controller-0.localdomain os-collect-config[7424]: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
5214:Mar 01 12:57:02 overcloud-controller-0.localdomain os-collect-config[7424]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
5817:Mar 01 12:57:42 overcloud-controller-0.localdomain galera(galera)[3738]: ERROR: MySQL is not running
5861:Mar 01 12:57:48 overcloud-controller-0.localdomain galera(galera)[4905]: ERROR: MySQL is not running
6115:Mar 01 12:58:14 overcloud-controller-0.localdomain os-collect-config[7424]: int::Base[storage_mgmt_vip-then-haproxy]/Exec[Creating order constraint storage_mgmt_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Colocation[control_vip-with-haproxy]/Pcmk_constraint[colo-ip-192.168.0.12-haproxy-clone]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[public_vip-then-haproxy]/Exec[Creating order constraint public_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[redis_vip-then-haproxy]/Exec[Creating order constraint redis_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[storage_vip-then-haproxy]/Exec[Creating order constraint storage_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: Finished catalog run in 35.84 seconds\u001b[0m\n", "deploy_stderr": "\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0}
6219:Mar 01 12:58:14 overcloud-controller-0.localdomain os-collect-config[7424]: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
6220:Mar 01 12:58:14 overcloud-controller-0.localdomain os-collect-config[7424]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
9454:Mar 01 13:02:23 overcloud-controller-0.localdomain os-collect-config[7424]: ]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/retry_interval]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Agent::Notification/Ceilometer_config[notification/ack_on_event_error]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/rpc_backend]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Alarm::Evaluator/Ceilometer_config[alarm/evaluation_service]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[oslo_messaging_rabbit/rabbit_hosts]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Agent::Central/Ceilometer_config[coordination/backend_url]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/http_timeout]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[DEFAULT/notification_topics]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Db/Ceilometer_config[database/max_retries]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Agent::Auth/Ceilometer_config[service_credentials/os_auth_url]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Logging/Ceilometer_config[DEFAULT/use_syslog]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Collector/Ceilometer_config[collector/udp_address]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Expirer/Ceilometer_config[database/time_to_live]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Alarm::Evaluator/Ceilometer_config[alarm/record_history]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Logging/Ceilometer_config[DEFAULT/use_stderr]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Alarm::Evaluator/Ceilometer_config[alarm/evaluation_interval]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[oslo_messaging_rabbit/rabbit_ha_queues]/ensure: created\u001
9489:Mar 01 13:02:23 overcloud-controller-0.localdomain os-collect-config[7424]: /swift/object-server/]/seluser: seluser changed 'unconfined_u' to 'system_u'\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy/Concat[/etc/swift/proxy-server.conf]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy/Concat[/etc/swift/proxy-server.conf]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments.concat]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy/Concat[/etc/swift/proxy-server.conf]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments.concat.out]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy/Concat[/etc/swift/proxy-server.conf]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy/Concat::Fragment[swift_proxy]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/00_swift_proxy]/ensure: defined content as '{md5}d40310054460199f880f40207d9a92f4'\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy::Catch_errors/Concat::Fragment[swift_catch_errors]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/24_swift_catch_errors]/ensure: defined content as '{md5}e07f0e5b125db7d6c8b4724c1648bcd5'\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy::Healthcheck/Concat::Fragment[swift_healthcheck]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/25_swift_healthcheck]/ensure: defined content as '{md5}8c92056c41082619d179f88ea15c5fc6'\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy::Cache/Concat::Fragment[swift_cache]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/23_swift_cache]/ensure: defined content as '{md5}20006a9d2864c3a4f5d6b1df19f09fce'\u001b[0m\n\u001b[mNotice: /Stage[main]/Swift::Proxy::Tempurl/Concat::Fragment[swift-proxy-tempurl]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/29_swift-proxy-tempurl]/ensure: defined content as '{md5}2fe004eae9f03fc684f9ed90044bd9c5'\u001b[0m\n\u001b[mNotice:
9630:Mar 01 13:02:23 overcloud-controller-0.localdomain os-collect-config[7424]: Notice: /Stage[main]/Ceilometer::Agent::Notification/Ceilometer_config[notification/ack_on_event_error]/ensure: created
10179:Mar 01 13:02:23 overcloud-controller-0.localdomain os-collect-config[7424]: Notice: /Stage[main]/Swift::Proxy::Catch_errors/Concat::Fragment[swift_catch_errors]/File[/var/lib/puppet/concat/_etc_swift_proxy-server.conf/fragments/24_swift_catch_errors]/ensure: defined content as '{md5}e07f0e5b125db7d6c8b4724c1648bcd5'
11233:Mar 01 13:03:31 overcloud-controller-0.localdomain crmd[30047]:  warning: Action 121 (openstack-heat-engine:0_start_0) on overcloud-controller-0 failed (target: 0 vs. rc: 7): Error
11235:Mar 01 13:03:31 overcloud-controller-0.localdomain crmd[30047]:  warning: Action 121 (openstack-heat-engine:0_start_0) on overcloud-controller-0 failed (target: 0 vs. rc: 7): Error
11807:Mar 01 13:03:45 overcloud-controller-0.localdomain crmd[30047]:  warning: Action 31 (neutron-server:0_monitor_0) on overcloud-controller-0 failed (target: 7 vs. rc: 0): Error
11809:Mar 01 13:03:45 overcloud-controller-0.localdomain crmd[30047]:  warning: Action 31 (neutron-server:0_monitor_0) on overcloud-controller-0 failed (target: 7 vs. rc: 0): Error
11810:Mar 01 13:03:45 overcloud-controller-0.localdomain crmd[30047]:  warning: Action 32 (httpd:0_monitor_0) on overcloud-controller-0 failed (target: 7 vs. rc: 0): Error
11811:Mar 01 13:03:45 overcloud-controller-0.localdomain crmd[30047]:  warning: Action 32 (httpd:0_monitor_0) on overcloud-controller-0 failed (target: 7 vs. rc: 0): Error
12032:Mar 01 13:03:48 overcloud-controller-0.localdomain object-server[8046]: Object audit (ZBF) "forever" mode completed: 0.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 0.00, Total bytes/sec: 0.00, Auditing time: 0.00, Rate: 0.00
12034:Mar 01 13:03:48 overcloud-controller-0.localdomain object-server[8047]: Object audit (ALL) "forever" mode completed: 0.00s. Total quarantined: 0, Total errors: 0, Total files/sec: 0.00, Total bytes/sec: 0.00, Auditing time: 0.00, Rate: 0.00
12160:Mar 01 13:03:50 overcloud-controller-0.localdomain proxy-server[8134]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb proxy-logging dlo versioned_writes proxy-server".
12171:Mar 01 13:03:50 overcloud-controller-0.localdomain proxy-server[8169]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb proxy-logging dlo versioned_writes proxy-server".
12175:Mar 01 13:03:50 overcloud-controller-0.localdomain proxy-server[8168]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb proxy-logging dlo versioned_writes proxy-server".
12185:Mar 01 13:03:50 overcloud-controller-0.localdomain proxy-server[8170]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb proxy-logging dlo versioned_writes proxy-server".
12189:Mar 01 13:03:50 overcloud-controller-0.localdomain proxy-server[8171]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck cache ratelimit tempurl formpost authtoken keystone staticweb proxy-logging dlo versioned_writes proxy-server".
13110:Mar 01 13:03:58 overcloud-controller-0.localdomain ceilometer-alarm-evaluator[9169]: ImportError: No module named redis
13111:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl [req-57b7f775-9ae2-42d9-acc8-480165ed44f7 - - - - -] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=type', 'list', 'Interface', 'int-br-ex'].
13112:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl Traceback (most recent call last):
13113:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl
13114:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl     log_fail_as_error=False).rstrip()
13115:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
13116:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl     raise RuntimeError(m)
13117:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl RuntimeError:
13118:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=type', 'list', 'Interface', 'int-br-ex']
13119:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl Exit code: 1
13120:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl
13121:Mar 01 13:03:58 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:58.548 9018 ERROR neutron.agent.ovsdb.impl_vsctl
13191:Mar 01 13:03:59 overcloud-controller-0.localdomain ceilometer-polling[9170]: ImportError: No module named redis
13194:Mar 01 13:03:59 overcloud-controller-0.localdomain neutron-openvswitch-agent[9018]: 2016-03-01 13:03:59.652 9018 ERROR neutron.agent.linux.ovsdb_monitor [req-57b7f775-9ae2-42d9-acc8-480165ed44f7 - - - - -] Interface monitor is not active
13209:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils [-] Invalid format of swift store configcfg
13210:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils Traceback (most recent call last):
13211:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/swift/utils.py", line 126, in _load_config
13212:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils     reference[param] = CONFIG.get(ref, param)
13213:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib64/python2.7/ConfigParser.py", line 618, in get
13214:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils     raise NoOptionError(option, section)
13215:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils NoOptionError: No option 'auth_address' in section: 'database'
13216:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.258 9020 ERROR glance_store._drivers.swift.utils
13217:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils [-] Invalid format of swift store configcfg
13218:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils Traceback (most recent call last):
13219:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/swift/utils.py", line 126, in _load_config
13220:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils     reference[param] = CONFIG.get(ref, param)
13221:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib64/python2.7/ConfigParser.py", line 618, in get
13222:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils     raise NoOptionError(option, section)
13223:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils NoOptionError: No option 'auth_address' in section: 'glance_store'
13224:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.268 9020 ERROR glance_store._drivers.swift.utils
13225:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils [-] Invalid format of swift store configcfg
13226:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils Traceback (most recent call last):
13227:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/swift/utils.py", line 126, in _load_config
13228:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils     reference[param] = CONFIG.get(ref, param)
13229:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib64/python2.7/ConfigParser.py", line 618, in get
13230:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils     raise NoOptionError(option, section)
13231:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils NoOptionError: No option 'auth_address' in section: 'image_format'
13232:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.269 9020 ERROR glance_store._drivers.swift.utils
13233:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils [-] Invalid format of swift store configcfg
13234:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils Traceback (most recent call last):
13235:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/swift/utils.py", line 126, in _load_config
13236:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils     reference[param] = CONFIG.get(ref, param)
13237:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils   File "/usr/lib64/python2.7/ConfigParser.py", line 618, in get
13238:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils     raise NoOptionError(option, section)
13239:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils NoOptionError: No option 'auth_address' in section: 'keystone_authtoken'
13240:Mar 01 13:04:00 overcloud-controller-0.localdomain glance-api[9020]: 2016-03-01 13:04:00.270 9020 ERROR glance_store._drivers.swift.utils
Comment 5 Zane Bitter 2016-03-01 15:41:46 EST
The failure was in the ComputeAllNodesValidationDeployment script on Compute node 0, so to diagnose the problem we'd need to see the output of that. You can get it from the Heat API with the command:

  heat deployment-show `heat resource-show 43cf7036-2b47-4680-b243-9ec7091976cf 0 | grep physical_resource_id | cut -d'|' -f3`
Comment 6 bigswitch 2016-03-01 16:00:31 EST
hi
this is the output


[stack@c5220-01 sosreport2]$ heat deployment-show `heat resource-show 43cf7036-2b47-4680-b243-9ec7091976cf 0 | grep physical_resource_id | cut -d'|' -f3`
{
  "status": "FAILED",
  "server_id": "2367a924-bff0-4e8a-befc-1c3e01a85dd5",
  "config_id": "49025b5d-036e-49b4-8516-02cd00cfae3b",
  "output_values": {
    "deploy_stdout": "Trying to ping 172.16.0.4 for local network 172.16.0.0/24...FAILURE\n",
    "deploy_stderr": "172.16.0.4 is not pingable. Local Network: 172.16.0.0/24\n",
    "deploy_status_code": 1
  },
  "creation_time": "2016-03-01T17:54:54",
  "updated_time": "2016-03-01T17:55:35",
  "input_values": {},
  "action": "CREATE",
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
  "id": "016747e7-896b-448b-a718-352dcafbac48"
}
Comment 7 bigswitch 2016-03-01 16:05:52 EST
hi
this is the output


[stack@c5220-01 sosreport2]$ heat deployment-show `heat resource-show 43cf7036-2b47-4680-b243-9ec7091976cf 0 | grep physical_resource_id | cut -d'|' -f3`
{
  "status": "FAILED",
  "server_id": "2367a924-bff0-4e8a-befc-1c3e01a85dd5",
  "config_id": "49025b5d-036e-49b4-8516-02cd00cfae3b",
  "output_values": {
    "deploy_stdout": "Trying to ping 172.16.0.4 for local network 172.16.0.0/24...FAILURE\n",
    "deploy_stderr": "172.16.0.4 is not pingable. Local Network: 172.16.0.0/24\n",
    "deploy_status_code": 1
  },
  "creation_time": "2016-03-01T17:54:54",
  "updated_time": "2016-03-01T17:55:35",
  "input_values": {},
  "action": "CREATE",
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1",
  "id": "016747e7-896b-448b-a718-352dcafbac48"
}
Comment 8 bigswitch 2016-03-01 16:06:36 EST
Ok I miss adding tenantNeworkVlanID to controller.yaml file. Will add it and redeploy
Comment 9 bigswitch 2016-03-02 03:20:04 EST
Found one problem. Looks like beta7 is not using the stable/liberty tripleo-heat-templates

For example the upstream stable/liberty has following code:
if 'cisco_n1kv' in hiera('neutron::plugins::ml2::mechanism_drivers') {
at https://github.com/openstack/tripleo-heat-templates/blob/stable/liberty/puppet/manifests/overcloud_compute.pp#L123

However, the actual beta7 code is like following:
if 'cisco_n1kv' in hiera('neutron_mechanism_drivers') {
Comment 10 Mike Burns 2016-03-04 08:44:00 EST
(In reply to bigswitch from comment #9)
> Found one problem. Looks like beta7 is not using the stable/liberty
> tripleo-heat-templates
> 
> For example the upstream stable/liberty has following code:
> if 'cisco_n1kv' in hiera('neutron::plugins::ml2::mechanism_drivers') {
> at
> https://github.com/openstack/tripleo-heat-templates/blob/stable/liberty/
> puppet/manifests/overcloud_compute.pp#L123
> 
> However, the actual beta7 code is like following:
> if 'cisco_n1kv' in hiera('neutron_mechanism_drivers') {

This is both true and not true.  The build included in beta 7 was openstack-tripleo-heat-templates-0.8.7-12.el7ost.  Since the 0.8.7 release upstream, upstream has moved along.  The change that you're seeing was done in this review:

https://review.openstack.org/#/c/269898/

The plan is that beta 8 will be on the 0.8.9 release which just happened this week and will contain that patch.

(In reply to bigswitch from comment #8)
> Ok I miss adding tenantNeworkVlanID to controller.yaml file. Will add it and
> redeploy

Did this help and resolve your problem?
Comment 11 bigswitch 2016-03-04 11:58:04 EST
Thanks a lot Mike. Please close this bug :)

Note You need to log in before you can comment on or make changes to this bug.