Created attachment 1072460 [details]
Overcloud deployment log

Description of problem:
On a overcloud deployment either hangs forever or it fails with the error

Resource CREATE failed: Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1


Version-Release number of selected component (if applicable):
poodles for 7.1 director

How reproducible:
Very all. the poodles being tested now 10/Sep - 11/Sep are in that state  

Steps to Reproduce:
1. Do an openstack overcloud deploy --debug --log-file overcloud_deployment_83.log --templates \\ --libvirt-type=qemu --neutron-network-type gre --neutron-tunnel-types gre \\ --ntp-server 10.5.26.10 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml


Actual results:
DEBUG: heatclient.common.http curl -g -i -X GET -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -H 'X-Auth-Url: http://192.0.2.1:5000/v2.0' -H 'Accept: application/json' -H 'X-Auth-Token: {SHA1}1a9cd13f082bbeaa129fb55702796c1de269fcb3' http://192.0.2.1:8004/v1/b8bdcf47da644ee8bff53c42c8930fb0/stacks/overcloud/a354755c-e823-403a-9531-328f879a0811
DEBUG: heatclient.common.http 
HTTP/1.1 200 OK
date: Fri, 11 Sep 2015 04:02:53 GMT
connection: keep-alive
content-type: application/json; charset=UTF-8
content-length: 10867
x-openstack-request-id: req-5db2f9c6-fdb7-4c57-91ec-7305d8830ba6

{"stack": {"parent": null, "disable_rollback": true, "description": "Nova API,Keystone,Heat Engine and API,Glance,Neutron,Dedicated MySQL server,Dedicated RabbitMQ Server,Group of Nova Computes\n", "links": [{"href": "http://192.0.2.1:8004/v1/b8bdcf47da644ee8bff53c42c8930fb0/stacks/overcloud/a354755c-e823-403a-9531-328f879a0811", "rel": "self"}], "stack_status_reason": "
Resource CREATE failed: Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1
", "stack_name": "overcloud", "stack_user_project_id": "a65bdbe25a19430eaa45a2949533f8b9", "stack_owner": "admin", "creation_time": "2015-09-11T03:53:39Z", "capabilities": [], "notification_topics": [], "updated_time": null, "timeout_mins": 240, "stack_status": "CREATE_FAILED", "parameters": {"NeutronPublicInterfaceRawDevice": "", "KeystoneSigningKey": "******", "NtpServer": "10.5.26.10", "ControllerCount": "1", "NeutronDhcpAgentsPerNetwork": "3", "NeutronFlatNetworks": "datacentre", "NovaEnableRbdBackend": "True", "controllerImage": "overcloud-full", "GlancePassword": "******", "NeutronTunnelIdRanges": "1:1000", "CinderEnableRbdBackend": "True", "SwiftReplicas": "3", "NeutronPublicInterfaceTag": "", "ControllerHostnameFormat": "%stackname%-controller-%index%", "CephClientKey": "", "KeystoneSigningCertificate": "-----BEGIN CERTIFICATE-----\nMIIDJDCCAgygAwIBAgIBAjANBgkqhkiG9w0BAQUFADBTMQswCQYDVQQGEwJYWDEO\nMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEU\nMBIGA1UEAxMLS2V5c3RvbmUgQ0EwHhcNMTUwOTExMDM1MzM4WhcNMjUwOTA4MDM1\nMzM4WjBYMQswCQYDVQQGEwJYWDEOMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVu\nc2V0MQ4wDAYDVQQKEwVVbnNldDEZMBcGA1UEAxMQS2V5c3RvbmUgU2lnbmluZzCC\nASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ3XnaoySaHwwWbdsU4tMB2H\nr8aFT9vypw9EWHJffrUVyHxezkzJjKgGUUhPnjIfH26vKGCwcKOztomIkPB4H8ib\nVrDAYqAV5wIyQB9qHz75fBAmw6Fh/Jv5j4k4kr1PUdeTY/4uuNtSzgtN9dB/F0yF\noWVUfIc7jt3U4EeqGs1jRD4yMAysJvTEAYZljRpcNgYjtp58P0CeM2rT+y/mpoZ6\nNlvUFsIfKSFX/crr3xiB0P/U2VlvqyKbf+I5VWgBILV91g93dVw6oB6CuuXUs5ev\nqkWPT7TKNBYNppsTsX9tDzXMW6a6PZojF/vNI5MlaQq8hq/HoXK89q7JJiYvL4cC\nAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAdCuHbSTIsje7gQL2hb7UphRr003xERLe\nYpHjpT/2eGUXuWQNeMvJFKFl7ElQ/cSsGVzykj8Tv6cRPOFJxnalcLNiUbFMMAsP\n2UZzksfuC4CwCVWQd85Yjz8NjLROv5iOvlzrXMURwDK66tZ4+IpYimot9LURjtpC\nz7ZrC5vLFBqCWN8AZqBRmUtH5u8GZ0YqRKHn5F7mK0QVGzX4uY5giyIwA0iu+L0w\njw5koNw7q3rahotOBhSX21XBnX/ibNC1J45EHDrxX+9J/ugp33Su8LDCMKJq+Nvh\nbacojcvcAZIBKCCzZAK/++KNgTei3yHhlGMGG7gaoglIJif5jIH2gg==\n-----END CERTIFICATE-----\n", "CephStorageCount": "1", "ServiceNetMap": "{\"NovaVncProxyNetwork\": \"internal_api\", \"CinderApiNetwork\": \"internal_api\", \"NovaApiNetwork\": \"internal_api\", \"CeilometerApiNetwork\": \"internal_api\", \"CephStorageHostnameResolveNetwork\": \"storage\", \"SwiftMgmtNetwork\": \"storage_mgmt\", \"MemcachedNetwork\": \"internal_api\", \"RabbitMqNetwork\": \"internal_api\", \"KeystoneAdminApiNetwork\": \"internal_api\", \"SwiftProxyNetwork\": \"storage\", \"NeutronTenantNetwork\": \"tenant\", \"CephClusterNetwork\": \"storage_mgmt\", \"NovaMetadataNetwork\": \"internal_api\", \"ControllerHostnameResolveNetwork\": \"internal_api\", \"NeutronApiNetwork\": \"internal_api\", \"GlanceApiNetwork\": \"storage\", \"ObjectStorageHostnameResolveNetwork\": \"internal_api\", \"KeystonePublicApiNetwork\": \"internal_api\", \"HeatApiNetwork\": \"internal_api\", \"GlanceRegistryNetwork\": \"internal_api\", \"RedisNetwork\": \"internal_api\", \"MysqlNetwork\": \"internal_api\", \"BlockStorageHostnameResolveNetwork\": \"internal_api\", \"ComputeHostnameResolveNetwork\": \"internal_api\", \"CephPublicNetwork\": \"storage\", \"MongoDbNetwork\": \"internal_api\", \"HorizonNetwork\": \"internal_api\", \"CinderIscsiNetwork\": \"storage\"}", "SnmpdReadonlyUserName": "ro_snmp_user", "KeyName": "default", "GlanceProtocol": "http", "CinderISCSIHelper": "lioadm", "BlockStorageImage": "overcloud-full", "OvercloudControlFlavor": "baremetal", "CloudName": "overcloud", "RabbitPassword": "******", "OS::stack_id": "a354755c-e823-403a-9531-328f879a0811", "PublicVirtualInterface": "br-ex", "Debug": "True", "NeutronAgentMode": "dvr_snat", "GlanceFilePcmkDevice": "", "OvercloudSwiftStorageFlavor": "baremetal", "SwiftPassword": "******", "SSLCertificate": "******", "CephStorageRemovalPolicies": "[]", "GlancePort": "9292", "SwiftPartPower": "10", "KeystoneNotificationFormat": "basic", "NeutronPublicInterfaceDefaultRoute": "", "HeatPassword": "******", "SwiftMountCheck": "False", "ControlVirtualInterface": "br-ex", "HeatStackDomainAdminPassword": "******", "OS::stack_name": "overcloud", "OvercloudComputeFlavor": "baremetal", "NovaRbdPoolName": "vms", "CephAdminKey": "AQBBUPJVAAAAABAAFHSwMgNe4KgFSC8cgVkALg==", "CinderEnableIscsiBackend": "False", "KeystoneNotificationDriver": "messaging", "ComputeHostnameFormat": "%stackname%-compute-%index%", "controllerExtraConfig": "{}", "NeutronMetadataProxySharedSecret": "unset", "ControllerEnableCephStorage": "False", "ControllerRemovalPolicies": "[]", "GlanceBackend": "rbd", "RabbitClientUseSSL": "False", "NeutronAllowL3AgentFailover": "False", "GlanceFilePcmkOptions": "", "RabbitUserName": "guest", "CephClientUserName": "openstack", "GlanceLogFile": "", "NeutronExternalNetworkBridge": "br-ex", "CeilometerBackend": "mongodb", "SwiftHashSuffix": "******", "RabbitCookieSalt": "unset", "AdminPassword": "******", "NeutronDVR": "False", "ExtraConfig": "{}", "CephStorageHostnameFormat": "%stackname%-cephstorage-%index%", "SSLKey": "******", "CephStorageImage": "overcloud-full", "NeutronDnsmasqOptions": "dhcp-option-force=26,1400", "MysqlMaxConnections": "1024", "GlanceRbdPoolName": "images", "NeutronTunnelTypes": "gre", "SSLCACertificate": "", "GlanceFilePcmkManage": "False", "PublicVirtualFixedIPs": "[]", "NeutronPassword": "******", "NeutronBridgeMappings": "datacentre:br-ex", "NeutronPublicInterface": "nic1", "NeutronVniRanges": "1:1000", "PublicVirtualNetwork": "ctlplane", "NovaImage": "overcloud-full", "NeutronEnableTunnelling": "True", "GlanceFilePcmkFstype": "nfs", "CephExternalMonHost": "", "OvercloudCephStorageFlavor": "baremetal", "CinderPassword": "******", "NeutronTypeDrivers": "vxlan,vlan,flat,gre", "ObjectStorageCount": "0", "CinderLVMLoopDeviceSize": "5000", "BlockStorageHostnameFormat": "%stackname%-blockstorage-%index%", "CinderNfsServers": "", "ImageUpdatePolicy": "REBUILD_PRESERVE_EPHEMERAL", "MysqlInnodbBufferPoolSize": "0", "BlockStorageCount": "0", "ControlFixedIPs": "[]", "AdminToken": "******", "SwiftMinPartHours": "1", "KeystoneSSLCertificate": "", "SwiftStorageImage": "overcloud-full", "NeutronL3HA": "False", "NovaComputeLibvirtType": "qemu", "EnableGalera": "True", "CephMonKey": "AQBBUPJVAAAAABAALJGSrIkKblNF79RRepCLlA==", "ComputeRemovalPolicies": "[]", "NeutronCorePlugin": "ml2", "HypervisorNeutronPhysicalBridge": "br-ex", "CeilometerMeteringSecret": "******", "CinderEnableNfsBackend": "False", "OS::project_id": "b8bdcf47da644ee8bff53c42c8930fb0", "HypervisorNeutronPublicInterface": "nic1", "NovaComputeDriver": "libvirt.LibvirtDriver", "CephClusterFSID": "aec9d49a-5838-11e5-923d-525400e04825", "CinderRbdPoolName": "volumes", "SnmpdReadonlyUserPassword": "******", "OvercloudBlockStorageFlavor": "baremetal", "BlockStorageRemovalPolicies": "[]", "RabbitClientPort": "5672", "NeutronComputeAgentMode": "dvr", "ComputeCount": "1", "KeystoneSSLCertificateKey": "******", "NeutronControlPlaneID": "7dbef0f0-bba5-4794-99c7-e4a37dfe773e", "EnablePacemaker": "False", "NeutronNetworkType": "gre", "CeilometerPassword": "******", "ObjectStorageRemovalPolicies": "[]", "NeutronMechanismDrivers": "openvswitch", "ControllerEnableSwiftStorage": "True", "GlanceNotifierStrategy": "noop", "KeystoneCACertificate": "-----BEGIN CERTIFICATE-----\nMIIDNzCCAh+gAwIBAgIBATANBgkqhkiG9w0BAQUFADBTMQswCQYDVQQGEwJYWDEO\nMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVuc2V0MQ4wDAYDVQQKEwVVbnNldDEU\nMBIGA1UEAxMLS2V5c3RvbmUgQ0EwHhcNMTUwOTExMDM1MzM3WhcNMjUwOTA4MDM1\nMzM3WjBTMQswCQYDVQQGEwJYWDEOMAwGA1UECBMFVW5zZXQxDjAMBgNVBAcTBVVu\nc2V0MQ4wDAYDVQQKEwVVbnNldDEUMBIGA1UEAxMLS2V5c3RvbmUgQ0EwggEiMA0G\nCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCyj1Oi/um3VK/fBv7LrT39utsSnt5v\nv4saEppzLfhV/WmNIiWzLoLq8QOBsaI1xM3KgCvddF2VsxmnSEoqvoML9lYdZm6t\nxnr4rgQ3uHW2evSDPtnL3atuWm/PbgkKX7cLdE3KRnkFODvr4lLmdIeWqtc6R6OH\n3mczyr1BXytN2bVfdGrj8vDilp3K6CVG/1P+vjAlmlPrX7AxAPk4T3fxWBYYCicG\n95t73dBcQcQHbkWWSx3o63vsh4VKoWr38junIawLZntqjlxzB7BWdsz3jZBiJ9kz\nlt/zbVuvmZ+Im0sCeutUFizO5+vL7wA61slYbV9C4nRcOBQQYfFQZnw1AgMBAAGj\nFjAUMBIGA1UdEwEB/wQIMAYBAf8CAQAwDQYJKoZIhvcNAQEFBQADggEBAGpACQl1\nPvomQfgdwLi8UEV/9VhCKKl3VFOG3PmsMh9clFR6XgpKQKdacp1l5M2FEvt7K804\nDoE0cBhMXTr6AFNiK3CN3XrmlhGoDKkCa/vdByAlNL+gELixXwoKYJVred3zz6YA\n5gmx2vqFPiUMlmhkr3VgOlQFiuQC15+QIg1lLk+Cwa800Lz1MovodqBVUyeAq8PA\nFj4tRx0q+SePAgEuzxS7i5lcWpnNrcGNQ9wjb3XyEFEsrVmOQAMkVHQ2g08fu7fx\nn9rAmKz5c0AKyG7P9zMtG1m9PTG3v8uT9l27BjSeVPz3wLSux9Z05cRHBukokR8G\nzIKWOyXInipUMb4=\n-----END CERTIFICATE-----\n", "NovaPassword": "******", "CinderNfsMountOptions": "", "NeutronPublicInterfaceIP": "", "ObjectStorageHostnameFormat": "%stackname%-objectstorage-%index%", "CeilometerComputeAgent": "", "NeutronServicePlugins": "router", "UpdateIdentifier": "", "NovaComputeExtraConfig": "{}", "NeutronNetworkVLANRanges": "datacentre:1:1000"}, "id": "a354755c-e823-403a-9531-328f879a0811", "outputs": [{"output_value": "192.0.2.6", "description": "VIP for Ceilometer API internal endpoint", "output_key": "CeilometerInternalVip"}, {"output_value": "192.0.2.6", "description": "VIP for Glance API internal endpoint", "output_key": "GlanceInternalVip"}, {"output_value": "192.0.2.6", "description": "VIP for Heat API internal endpoint", "output_key": "HeatInternalVip"}, {"output_value": "192.0.2.6", "description": "VIP for Keystone API internal endpoint", "output_key": "KeystoneInternalVip"}, {"output_value": "192.0.2.8", "description": "Controller VIP for public API endpoints", "output_key": "PublicVip"}, {"output_value": "http://192.0.2.8:5000/v2.0/", "description": "URL for the Overcloud Keystone service", "output_key": "KeystoneURL"}, {"output_value": "192.0.2.6", "description": "VIP for Neutron API internal endpoint", "output_key": "NeutronInternalVip"}, {"output_value": "192.0.2.6", "description": "VIP for Swift Proxy internal endpoint", "output_key": "SwiftInternalVip"}, {"output_value": "192.0.2.6", "description": "VIP for Cinder API internal endpoint", "output_key": "CinderInternalVip"}, {"output_value": "192.0.2.6", "description": "Keystone Admin VIP endpoint", "output_key": "KeystoneAdminVip"}, {"output_value": "192.0.2.6", "description": "VIP for Nova API internal endpoint", "output_key": "NovaInternalVip"}], "template_description": "Nova API,Keystone,Heat Engine and API,Glance,Neutron,Dedicated MySQL server,Dedicated RabbitMQ Server,Group of Nova Computes\n"}}

ERROR: openstack Heat Stack create failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 1208, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 520, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 449, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.
DEBUG: openstackclient.shell clean_up DeployOvercloud
DEBUG: openstackclient.shell got an error: Heat Stack create failed.
ERROR: openstackclient.shell Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run
    return super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 1208, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 520, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 449, in _heat_deploy
    raise Exception("Heat Stack create failed.")
Exception: Heat Stack create failed.

Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
Stack failed with status: Resource CREATE failed: Error: resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1


Expected results:
Deploy success

Additional info: