Bug 1251156

Description of problem:
Overcloud heat stack fails to deploy on baremetal (DELL) with 3 controllers and 1 compute. The failure is always during the step6 on the 1st controller node (0) on puppet apply.

Version-Release number of selected component (if applicable):
rhos-release-0.65-1.noarch
python-rdomanager-oscplugin-0.0.8-43.el7ost.noarch
puddle images 2015-07-30.1

How reproducible:
Randomly (8/10)

Steps to Reproduce:
1. Deploy undercloud with default configuration
2. Deploy overcloud with default configuration on 3 controllers and 1 compute

[stack@instack ~]$ openstack overcloud deploy --plan overcloud --control-scale 3 --compute-scale 1 --ceph-storage-scale 0 --block-storage-scale 0 --swift-storage-scale 0


Actual results:

[stack@instack ~]$ heat resource-list -n5 overcloud |grep FAILED
| ControllerNodesPostDeployment               | 12aec09d-c327-40b2-9f03-bd78175c6540          | OS::TripleO::ControllerPostDeployment             | CREATE_FAILED   | 2015-08-05T12:47:58Z |                                             |
| ControllerOvercloudServicesDeployment_Step6 | 10da2de7-a9dd-4ee1-9377-7591f636f87c          | OS::Heat::StructuredDeployments                   | CREATE_FAILED   | 2015-08-05T13:02:14Z | ControllerNodesPostDeployment               |
| 0                                           | 37f6d34b-7e5b-4849-a483-dc9377f07bcf          | OS::Heat::StructuredDeployment                    | CREATE_FAILED   | 2015-08-05T13:14:17Z | ControllerOvercloudServicesDeployment_Step6 |

[stack@instack ~]$ heat deployment-output-show 37f6d34b-7e5b-4849-a483-dc9377f07bcf deploy_stderr 
Warning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.
Warning: Scope(Swift::Storage::Server[6002]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.
Warning: Scope(Swift::Storage::Server[6002]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.
Warning: Scope(Swift::Storage::Server[6001]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.
Warning: Scope(Swift::Storage::Server[6001]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.
Warning: Scope(Swift::Storage::Server[6000]): The default incoming_chmod set to 0644 may yield in error prone directories and will be changed in a later release.
Warning: Scope(Swift::Storage::Server[6000]): The default outgoing_chmod set to 0644 may yield in error prone directories and will be changed in a later release.
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')
Error: heat-keystone-setup-domain returned 1 instead of one of [0]
Error: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns: change from notrun to 0 failed: heat-keystone-setup-domain returned 1 instead of one of [0]
Warning: /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id]: Skipping because of failed dependencies

[stack@instack ~]$ heat deployment-output-show 37f6d34b-7e5b-4849-a483-dc9377f07bcf deploy_stdout
Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 17.66 seconds
Notice: /Stage[main]/Main/Exec[galera-ready]/returns: executed successfully
Notice: /Stage[main]/Main/Exec[neutron-server-start-wait-stop]/returns: executed successfully
Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]/ensure: created
Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]/ensure: created
Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]/ensure: ensure changed 'running' to 'stopped'
Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: created
Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[services]/ensure: created
Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: created
Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]/ensure: created
Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]/roles: roles changed ['_member_'] to 'admin'
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/bin/heat-keystone-setup-domain", line 156, in <module>
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     main()
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/bin/heat-keystone-setup-domain", line 137, in main
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     description="Heat domain admin")
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     return func(*args, **kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/v3/users.py", line 66, in create
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     domain_id=base.getid(domain),
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 41, in getid
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     if obj.uuid:
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/openstack/common/apiclient/base.py", line 477, in __getattr__
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     self.get()
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/openstack/common/apiclient/base.py", line 495, in get
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     new = self.manager.get(self.id)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/v3/domains.py", line 47, in get
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     domain_id=base.getid(domain))
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 73, in func
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     return f(*args, **new_kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 334, in get
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     self.key)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 136, in _get
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     resp, body = self.client.get(url, **kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 170, in get
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     return self.request(url, 'GET', **kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     return self.session.request(url, method, **kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     return func(*args, **kwargs)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 397, in request
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns:     raise exceptions.from_response(resp, method, url)
Notice: /Stage[main]/Heat::Keystone::Domain/Exec[heat_domain_create]/returns: keystoneclient.openstack.common.apiclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-b59900b3-08a7-4ea3-a875-9b9461f80c3b)
Notice: /Stage[main]/Heat::Keystone::Domain/Heat_domain_id_setter[heat_domain_id]: Dependency Exec[heat_domain_create] has failures: true
Notice: /Stage[main]/Pacemaker::Corosync/Exec[enable-not-start-tripleo_cluster]/returns: executed successfully
Notice: /Stage[main]/Pacemaker::Corosync/Exec[Set password for hacluster user on tripleo_cluster]/returns: executed successfully
Notice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: executed successfully
Notice: Pacemaker has reported quorum achieved
Notice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/message: defined 'message' as 'Pacemaker has reported quorum achieved'
Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Heat::Api/Service[heat-api]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Triggered 'refresh' from 2 events
Notice: Finished catalog run in 74.81 seconds