Bug 1261192

Summary: heat stack-update failed when attempting to deploy three additional compute nodes
Product: Red Hat OpenStack Reporter: bigswitch <rhosp-bugs-internal>
Component: rhosp-directorAssignee: chris alfonso <calfonso>
Status: CLOSED INSUFFICIENT_DATA QA Contact: yeylon <yeylon>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: hbrock, mburns, rhel-osp-director-maint, rhosp-bugs-internal, srevivo
Target Milestone: ---   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-30 16:52:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description bigswitch 2015-09-08 20:34:58 UTC
Description of problem:
After successfully deploying three controller and two compute nodes, I attempt to install three more compute nodes. heat stack update failed with message 

"ResourceUnknownStatus: Resource failed - Unknown status FAILED due to \"ResourceUnknownStatus: Resource failed - Unknown status FAILED due to \"Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1\"\"",

ironic shows the nodes are deployed, and nova list shows deployed node as active


[stack@rhosp7-undercloud ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| a3106bce-f5d4-409e-b2a6-1a38016587ef | overcloud  | UPDATE_FAILED | 2015-09-08T19:25:50Z |
+--------------------------------------+------------+---------------+----------------------+
[stack@rhosp7-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+----------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks             |
+--------------------------------------+------------------------+--------+------------+-------------+----------------------+
| 0df1cf5c-61df-4679-b791-c1b4062fc483 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.156 |
| 41e4d6d1-05a5-4971-a394-41eb48833eb0 | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.0.2.158 |
| d7113ec2-e1dc-4e85-a29c-652f631dd928 | overcloud-compute-2    | ACTIVE | -          | Running     | ctlplane=192.0.2.161 |
| 9d4e31dd-6134-464e-b247-3e6c1da72a32 | overcloud-compute-3    | ACTIVE | -          | Running     | ctlplane=192.0.2.162 |
| 98b7fde0-7f0c-4e6f-aca8-117e1cad0ff6 | overcloud-compute-4    | ACTIVE | -          | Running     | ctlplane=192.0.2.160 |
| 38dc490f-184d-4e7a-a113-4c0b96729ee5 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.157 |
| a191001c-31ae-40cf-877e-30a6c4292d62 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.155 |
| 48d9d33d-265d-4483-a576-d656b335993e | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.159 |
+--------------------------------------+------------------------+--------+------------+-------------+----------------------+
[stack@rhosp7-undercloud ~]$ ironic node-list
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| 510b4ab5-32a6-4d6f-b591-c65fbc17d62c | None | 48d9d33d-265d-4483-a576-d656b335993e | power on    | active          | False       |
| bea1652a-0be8-4282-8ac1-76f01ea87856 | None | a191001c-31ae-40cf-877e-30a6c4292d62 | power on    | active          | False       |
| 9a1dde2f-994f-40ca-8460-74b98a8bcfee | None | 38dc490f-184d-4e7a-a113-4c0b96729ee5 | power on    | active          | False       |
| 791665f4-e610-4453-8cdd-cb2fb6def37a | None | None                                 | power off   | available       | False       |
| 9cb73c6e-0a91-475c-87e9-f9a6174506db | None | None                                 | power off   | available       | False       |
| e72db0ca-a1fb-4506-a561-058286b85a22 | None | None                                 | power off   | available       | False       |
| 9ea548a0-3d95-401e-a571-198f527de340 | None | None                                 | power off   | available       | False       |
| 575cad27-74e8-45da-8510-48d44b927997 | None | None                                 | power off   | available       | False       |
| 04241fc2-84d5-4e81-87a4-e78189afc5a0 | None | None                                 | power off   | available       | False       |
| 85594939-c3e7-48c4-bcf2-63b39d0333c1 | None | None                                 | power off   | available       | False       |
| a10d98c1-c916-4761-83e1-426441002d18 | None | None                                 | power off   | available       | False       |
| f14bf44c-a5db-40ac-ac12-99d5c825866e | None | None                                 | power off   | available       | False       |
| ffcaa4fd-1f6c-4842-8f9d-510f5aada750 | None | 98b7fde0-7f0c-4e6f-aca8-117e1cad0ff6 | power on    | active          | False       |
| 5ad905c2-94c2-4c8f-8ac6-78cbaa292c58 | None | None                                 | power off   | available       | False       |
| 20df4168-7196-49d8-9f75-afd9d5715911 | None | 9d4e31dd-6134-464e-b247-3e6c1da72a32 | power on    | active          | False       |
| a8c647dc-5e59-46ec-aede-072eb7d9038a | None | None                                 | power off   | available       | False       |
| 6b2b4529-fe04-451b-b0b9-7ebbd1ee0164 | None | None                                 | power off   | available       | False       |
| b7c696b8-df7f-48ff-abbb-4c8d599a4ef1 | None | d7113ec2-e1dc-4e85-a29c-652f631dd928 | power on    | active          | False       |
| de8c34ec-d916-41aa-89a9-bbfae0714e30 | None | None                                 | power off   | available       | False       |
| 1dac9659-260d-4536-9eef-683747d024d6 | None | None                                 | power off   | available       | False       |
| 0e31520f-7701-411f-9056-54ca4f0392d7 | None | None                                 | power off   | available       | False       |
| 7b787a96-a1f5-49de-927b-d060f2079fdf | None | None                                 | power off   | available       | False       |
| 88b7134f-9566-48e2-82f6-e31cc8ea9191 | None | None                                 | power off   | available       | False       |
| 5fc6892e-73c0-4443-89f4-c66bb0dc1600 | None | None                                 | power off   | available       | False       |
| 7d930c2e-b426-45eb-a685-d1b4bf8120fd | None | None                                 | power off   | available       | False       |
| 81b15a3b-fbac-4117-a5a6-6f3720bb25d9 | None | None                                 | power off   | available       | False       |
| 8b9d9cfe-c435-4bca-9eb5-ac77ae90007c | None | 0df1cf5c-61df-4679-b791-c1b4062fc483 | power on    | active          | False       |
| 54fc6c98-13bc-4ce6-a865-9fa092eacae8 | None | None                                 | power off   | available       | False       |
| 74d28dbe-e733-4d78-b721-193665b43230 | None | None                                 | power off   | available       | False       |
| f18bbc32-3350-4c3e-a81b-95773eb2096f | None | None                                 | power off   | available       | False       |
| 6e388e0c-60fc-4e1b-b2b6-22c70887927a | None | None                                 | power off   | available       | False       |
| 89098be7-486c-46a2-a489-51a246c74b7d | None | 41e4d6d1-05a5-4971-a394-41eb48833eb0 | power on    | active          | False       |
| 9d07697a-0221-4f1a-a1e7-808f9c0b8a46 | None | None                                 | power off   | available       | False       |
| 7402b4a8-0410-42fb-a1c0-5ea0cb50feef | None | None                                 | power off   | available       | False       |
| 681c83a4-250f-42b5-b662-3787d7a26270 | None | None                                 | power off   | available       | False       |
| aec9147a-cdd2-4dc1-8a0f-5672f438e5c6 | None | None                                 | power off   | available       | False       |
| 9083ae94-ce0b-4e03-8271-dc757178f5a7 | None | None                                 | power off   | available       | False       |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+



The resource was found at http://192.0.2.1:8004/v1/2adf80a11b894d9b8da5dbdc36d56ae4/stacks/overcloud/a3106bce-f5d4-409e-b2a6-1a38016587ef; you should be redirected automatically.

DEBUG: heatclient.common.http curl -g -i -X GET -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -H 'X-Auth-Url: http://192.0.2.1:5000/v2.0' -H 'Accept: application/json' -H 'X-Auth-Token: {SHA1}60a9e9bbb7c42bbb8de1b9784583c4339ad82df9' http://192.0.2.1:8004/v1/2adf80a11b894d9b8da5dbdc36d56ae4/stacks/overcloud/a3106bce-f5d4-409e-b2a6-1a38016587ef
DEBUG: heatclient.common.http
HTTP/1.1 200 OK
date: Tue, 08 Sep 2015 20:22:15 GMT
connection: keep-alive
content-type: application/json; charset=UTF-8
content-length: 10520
x-openstack-request-id: req-18f719ae-4577-4e75-9d4b-2f9e7a7ce7b6

{"stack": {"parent": null, "disable_rollback": true, "description": "Nova API,Keystone,Heat Engine and API,Glance,Neutron,Dedicated MySQL server,Dedicated RabbitMQ Server,Group of Nova Computes\n", "links": [{"href": "http://192.0.2.1:8004/v1/2adf80a11b894d9b8da5dbdc36d56ae4/stacks/overcloud/a3106bce-f5d4-409e-b2a6-1a38016587ef", "rel": "self"}], "stack_status_reason": "ResourceUnknownStatus: Resource failed - Unknown status FAILED due to \"ResourceUnknownStatus: Resource failed - Unknown status FAILED due to \"Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1\"\"", "stack_name": "overcloud", "stack_user_project_id": "0d17f8cba077495091c68e614790b7cc", "stack_owner": "admin", "creation_time": "2015-09-08T19:25:50Z", "capabilities": [], "notification_topics": [], "updated_time": "2015-09-08T20:07:44Z", "timeout_mins": 240, "stack_status": "UPDATE_FAILED", "parameters": {"NeutronPublicInterfaceRawDevice": "", "KeystoneSigningKey": "******", "NtpServer": "0.rhel.pool.ntp.org", "NeutronDhcpAgentsPerNetwork": "3", "NeutronFlatNetworks": "datacentre", "NovaEnableRbdBackend": "False", "controllerImage": "overcloud-full", "GlancePassword": "******", "NeutronTunnelIdRanges": "1:1000", "CinderEnableRbdBackend": "False", "SwiftReplicas": "3", "NeutronPublicInterfaceTag": "", "ControllerHostnameFormat": "%stackname%-controller-%index%", "KeystoneSigningCertificate": "-----BEGIN CERTIFICATE-----\nMIIDJDCCAgygAwIBAgIBAjANBgkqhkiG9w0BAQUFADBTMQswCQYDVQQGEwJYWDEO\nMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEU\nMBIGA1UEAxMLS2V5c3RvbmUgQ0EwHhcNMTUwOTA4MTkyNTQ2WhcNMjUwOTA1MTky\nNTQ2WjBYMQswCQYDVQQGEwJYWDEOMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVu\nc2V0MQ4wDAYDVQQKEwVVbnNldDEZMBcGA1UEAxMQS2V5c3RvbmUgU2lnbmluZzCC\nASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMe2VS/SsV+aSalFabbEfaZe\nlXB7iigQNoJ6IB4GROV80bPrXTCTb6tcfw0SXSyVQOmtdxlj9HHLf+PmXymPFI7J\nncna9KCgnEVIRxxy/Fnbh9ZyC+y5M92EmyU3LDs34S5fEDlLaynk5wnNVxq6SqR6\nR3JNy45qVMCPt3Bx6tNu/ckrihaEYmiUcilyDelHWDjEo1LTrw6AWd4xEbHfO5F5\nvttxKSFo71s7lK5hxMC6obPi6Wu001/UxaeMyCU9zeIYyW2zzVtN8dmnx5s7r5cB\nOAu61BRyNf1XIm8g5UmEtcAWgEW6XXfcDbaKfsx0L2LZVEpWZWNLf2+puwM72LMC\nAwEAATANBgkqhkiG9w0BAQUFAAOCAQEADTNwaOYIfvQj5kIjFQuTq5D2YLy5aswo\nJRDVzfrSUfk7eaJlFRtFOjk2zQFFkMQ5UfoWA7OBIuSPdqqlSzvkhozdlSTCesf8\nkGWiQ16vHpMlGqycIkJfin3FKxT32ety1UBFjosRg9tjXpZRLkfC2ved9X1h5QoK\nx2GIH1843PDayxKkZoXHr+cQeo2eX/HNCphO2na/U+59KqN/WOz+kNGu5aE6mkJA\ncu0K7S1iwXehgtLWXPBdU7rUx2W1HlTf+PB/FUS+CAFitByEldE4XGtu6ccxq9Ux\nuC8NEnzSIhLWBVbgbsLRvdv4qq6owSGIqLzmjG3Zb9UATAJ177wwQQ==\n-----END CERTIFICATE-----\n", "CephStorageCount": "0", "ServiceNetMap": "{\"NovaVncProxyNetwork\": \"internal_api\", \"CinderApiNetwork\": \"internal_api\", \"NovaApiNetwork\": \"internal_api\", \"CeilometerApiNetwork\": \"internal_api\", \"CephStorageHostnameResolveNetwork\": \"storage\", \"SwiftMgmtNetwork\": \"storage_mgmt\", \"MemcachedNetwork\": \"internal_api\", \"RabbitMqNetwork\": \"internal_api\", \"KeystoneAdminApiNetwork\": \"internal_api\", \"SwiftProxyNetwork\": \"storage\", \"NeutronTenantNetwork\": \"tenant\", \"CephClusterNetwork\": \"storage_mgmt\", \"NovaMetadataNetwork\": \"internal_api\", \"ControllerHostnameResolveNetwork\": \"internal_api\", \"NeutronApiNetwork\": \"internal_api\", \"GlanceApiNetwork\": \"storage\", \"ObjectStorageHostnameResolveNetwork\": \"internal_api\", \"KeystonePublicApiNetwork\": \"internal_api\", \"HeatApiNetwork\": \"internal_api\", \"GlanceRegistryNetwork\": \"internal_api\", \"RedisNetwork\": \"internal_api\", \"MysqlNetwork\": \"internal_api\", \"BlockStorageHostnameResolveNetwork\": \"internal_api\", \"ComputeHostnameResolveNetwork\": \"internal_api\", \"CephPublicNetwork\": \"storage\", \"MongoDbNetwork\": \"internal_api\", \"HorizonNetwork\": \"internal_api\", \"CinderIscsiNetwork\": \"storage\"}", "SnmpdReadonlyUserName": "ro_snmp_user", "KeyName": "default", "AdminToken": "******", "CinderISCSIHelper": "lioadm", "BlockStorageImage": "overcloud-full", "OvercloudControlFlavor": "control", "CloudName": "overcloud", "RabbitPassword": "******", "OS::stack_id": "a3106bce-f5d4-409e-b2a6-1a38016587ef", "PublicVirtualInterface": "br-ex", "Debug": "True", "NeutronPublicInterface": "nic1", "GlanceFilePcmkDevice": "", "OvercloudComputeFlavor": "compute", "CinderEnableIscsiBackend": "True", "NeutronMetadataProxySharedSecret": "unset", "CephStorageRemovalPolicies": "[]", "GlancePort": "9292", "SwiftPartPower": "10", "NeutronPublicInterfaceDefaultRoute": "", "HeatPassword": "******", "SwiftMountCheck": "False", "ControlVirtualInterface": "br-ex", "HeatStackDomainAdminPassword": "******", "OS::stack_name": "overcloud", "OvercloudSwiftStorageFlavor": "baremetal", "CephAdminKey": "", "SwiftPassword": "******", "ComputeHostnameFormat": "%stackname%-compute-%index%", "controllerExtraConfig": "{\"neutron_ovs_use_veth\": true, \"neutron::agents::dhcp::enable_metadata_network\": true, \"neutron::agents::dhcp::enable_isolated_metadata\": true}", "SSLCertificate": "******", "ControllerEnableCephStorage": "False", "ControllerRemovalPolicies": "[]", "GlanceBackend": "swift", "RabbitClientUseSSL": "False", "NeutronAllowL3AgentFailover": "False", "GlanceFilePcmkOptions": "", "RabbitUserName": "guest", "GlanceLogFile": "", "NeutronExternalNetworkBridge": "''", "BlockStorageHostnameFormat": "%stackname%-blockstorage-%index%", "SwiftStorageImage": "overcloud-full", "RabbitCookieSalt": "unset", "AdminPassword": "******", "NeutronDVR": "False", "CephClusterFSID": "", "CephStorageHostnameFormat": "%stackname%-cephstorage-%index%", "NeutronAgentMode": "dvr_snat", "NeutronEnableTunnelling": "False", "CephStorageImage": "overcloud-full", "NeutronDnsmasqOptions": "dhcp-option-force=26,1400", "MysqlMaxConnections": "1024", "ControllerCount": "3", "NeutronTunnelTypes": "gre", "SSLCACertificate": "", "GlanceFilePcmkManage": "False", "PublicVirtualFixedIPs": "[]", "NeutronPassword": "******", "NeutronBridgeMappings": "datacentre:br-ex", "NeutronVniRanges": "1:1000", "PublicVirtualNetwork": "ctlplane", "NovaImage": "overcloud-full", "SSLKey": "******", "GlanceFilePcmkFstype": "nfs", "OvercloudCephStorageFlavor": "baremetal", "CinderPassword": "******", "HypervisorNeutronPublicInterface": "nic1", "ObjectStorageCount": "0", "CinderLVMLoopDeviceSize": "5000", "CeilometerBackend": "mongodb", "CinderNfsServers": "", "ImageUpdatePolicy": "REBUILD_PRESERVE_EPHEMERAL", "MysqlInnodbBufferPoolSize": "0", "BlockStorageCount": "0", "ControlFixedIPs": "[]", "GlanceProtocol": "http", "SwiftHashSuffix": "******", "SwiftMinPartHours": "1", "KeystoneSSLCertificate": "", "NeutronL3HA": "True", "NovaComputeLibvirtType": "qemu", "EnableGalera": "True", "CephMonKey": "", "ComputeRemovalPolicies": "[]", "HypervisorNeutronPhysicalBridge": "br-ex", "CeilometerMeteringSecret": "******", "CinderEnableNfsBackend": "False", "OS::project_id": "2adf80a11b894d9b8da5dbdc36d56ae4", "NovaComputeDriver": "libvirt.LibvirtDriver", "ExtraConfig": "{}", "SnmpdReadonlyUserPassword": "******", "OvercloudBlockStorageFlavor": "baremetal", "BlockStorageRemovalPolicies": "[]", "RabbitClientPort": "5672", "NovaComputeExtraConfig": "{}", "ComputeCount": "2", "KeystoneSSLCertificateKey": "******", "NeutronControlPlaneID": "23e23f02-947c-455f-ad13-2309d36ad583", "EnablePacemaker": "False", "NeutronNetworkType": "vlan", "CeilometerPassword": "******", "ObjectStorageRemovalPolicies": "[]", "NeutronMechanismDrivers": "openvswitch,bsn_ml2", "ControllerEnableSwiftStorage": "True", "GlanceNotifierStrategy": "noop", "KeystoneCACertificate": "-----BEGIN CERTIFICATE-----\nMIIDNzCCAh+gAwIBAgIBATANBgkqhkiG9w0BAQUFADBTMQswCQYDVQQGEwJYWDEO\nMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEU\nMBIGA1UEAxMLS2V5c3RvbmUgQ0EwHhcNMTUwOTA4MTkyNTQ2WhcNMjUwOTA1MTky\nNTQ2WjBTMQswCQYDVQQGEwJYWDEOMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVu\nc2V0MQ4wDAYDVQQKEwVVbnNldDEUMBIGA1UEAxMLS2V5c3RvbmUgQ0EwggEiMA0G\nCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQClN71o2AKh2zY6hMP4yLaOhguigOoF\n1T7iotweb/59IIGbQG8aXc98iZfhICtLBUeyJw05xvSt30ISnwagHlDrJfm/1fJj\nquY9T1bPE6D8uplmS0ZVrAKln0YK6gcKXU18cFwtK8X1MWXOZuqT/8AHwTbjjUHD\nax66E2t92oJnuSU20U5LmnYX5XMkh6uzzP0ro1RAgJvGKCZv6bgAVgSg+Nz04cdv\nd3FZvLcb4sCvgnkxXvti2GurR3Q9jCNt/VMQy2i8DpJsT8dFpK+K8l2VXjcvb7wp\nygskaqF9zJwWcT5//ORm3C2UnauxcMHXVJE5TNmox8Fg3JkBiAqpkatBAgMBAAGj\nFjAUMBIGA1UdEwEB/wQIMAYBAf8CAQAwDQYJKoZIhvcNAQEFBQADggEBAIus9OrQ\nW9m6+vJhcjMBb4HDoGui9G94bAXiv/psjH2/zRZVHJyd0gJzaR/yug3JAETlAIk1\nGoV4/PwfylF/ZJocoPs33/m9ri2XDW7lOv8Xvrjk/fW6K+FF3VE54L9ldk7SjUuL\ntqc6amz4CacvCOEyvGaOg4fd4UCVfMe91fB0cb7m+JVfzdqV2aOYrwcfV8tK90uW\nsgQ4GlXsy560MbriLYNDRA/pMVYP36yUE00DjPKmzOK1nn9h3rpL0rXvnlDNFzrh\ns9/tuGsjsp+uQ9dJnSAXvSz/jwbz70leQcyjDa5HcKOrpeUNHVuA9ZIxrYL4pgfH\ntmLuZT1GnmQ3gYk=\n-----END CERTIFICATE-----\n", "NovaPassword": "******", "CinderNfsMountOptions": "", "NeutronPublicInterfaceIP": "", "ObjectStorageHostnameFormat": "%stackname%-objectstorage-%index%", "CeilometerComputeAgent": "", "UpdateIdentifier": "", "NeutronComputeAgentMode": "dvr", "NeutronNetworkVLANRanges": "datacentre:1001:2000"}, "id": "a3106bce-f5d4-409e-b2a6-1a38016587ef", "outputs": [{"output_value": "172.17.0.11", "description": "VIP for Ceilometer API internal endpoint", "output_key": "CeilometerInternalVip"}, {"output_value": "172.18.0.10", "description": "VIP for Glance API internal endpoint", "output_key": "GlanceInternalVip"}, {"output_value": "172.17.0.11", "description": "VIP for Heat API internal endpoint", "output_key": "HeatInternalVip"}, {"output_value": "172.17.0.11", "description": "VIP for Keystone API internal endpoint", "output_key": "KeystoneInternalVip"}, {"output_value": "10.17.66.11", "description": "Controller VIP for public API endpoints", "output_key": "PublicVip"}, {"output_value": "http://10.17.66.11:5000/v2.0/", "description": "URL for the Overcloud Keystone service", "output_key": "KeystoneURL"}, {"output_value": "172.17.0.11", "description": "VIP for Neutron API internal endpoint", "output_key": "NeutronInternalVip"}, {"output_value": "172.18.0.10", "description": "VIP for Swift Proxy internal endpoint", "output_key": "SwiftInternalVip"}, {"output_value": "172.17.0.11", "description": "VIP for Cinder API internal endpoint", "output_key": "CinderInternalVip"}, {"output_value": "172.17.0.11", "description": "VIP for Nova API internal endpoint", "output_key": "NovaInternalVip"}], "template_description": "Nova API,Keystone,Heat Engine and API,Glance,Neutron,Dedicated MySQL server,Dedicated RabbitMQ Server,Group of Nova Computes\n"}}

ERROR: openstack Heat Stack update failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 824, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 498, in _deploy_tripleo_heat_templates
    parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 443, in _heat_deploy
    raise Exception("Heat Stack update failed.")
Exception: Heat Stack update failed.
DEBUG: openstackclient.shell clean_up DeployOvercloud
DEBUG: openstackclient.shell got an error: Heat Stack update failed.
ERROR: openstackclient.shell Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run
    return super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 824, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 498, in _deploy_tripleo_heat_templates
    parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 443, in _heat_deploy
    raise Exception("Heat Stack update failed.")
Exception: Heat Stack update failed.

Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates

Version-Release number of selected component (if applicable):

deploy command is 

 openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/bigswitch-config.yaml --neutron-network-type vlan --neutron-network-vlan-ranges datacentre:1001:2000 --neutron-disable-tunneling --compute-scale 5 --control-scale 3 --ceph-storage-scale 0 --templates --control-flavor control --compute-flavor compute --ntp-server 0.rhel.pool.ntp.org --debug 2>&1 | tee haha.log



How reproducible:
This is the first time I am seeing this, typically it was failing when attempting to scale to 10 compute nodes.

Steps to Reproduce:
1. Deploy 3 controller and 2 compute, it is successful
2. Deploy another 3 more compute, making total to 5
3. Deployment failed

Actual results:


Expected results:


Additional info:

Comment 3 bigswitch 2015-09-08 20:53:26 UTC
ironic log and sosreport is uploaded to 

https://bigswitch.box.com/s/3hk3qd5d2izxlpa2tuo1buknm8gj76rq

Comment 4 chris alfonso 2015-09-14 16:49:09 UTC
Is this the same issue you asked about via email? Was it the keystone timeout issue, or max resources?

Comment 5 bigswitch 2015-09-14 20:25:54 UTC
I seen this only once, so its not a maximum resource issue or the keystone timeout issue. I'm not sure if this is reproducible though.

Comment 6 bigswitch 2015-09-14 20:26:08 UTC
I seen this only once, so its not a maximum resource issue or the keystone timeout issue. I'm not sure if this is reproducible though.

Comment 7 chris alfonso 2015-09-22 16:51:02 UTC
Are you able to reproduce this issue?

Comment 8 chris alfonso 2015-09-30 16:52:06 UTC
Haven't heard anything on this. Just wanted to check to see if you're still running into this issue with a scale up and I saw you're able to scale to 50 so I'm going to close this bug. If it's still an issue, you can reopen.

Comment 9 Red Hat Bugzilla 2023-09-14 03:05:02 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days