Description of problem: When scaling to 300 + baremetal node deployment fails. This is a perf-scale lab effort to scale to 500 + baremetal nodes Version-Release number of selected component (if applicable): How reproducible: Verified 2/2 times Steps to Reproduce: 1. Deploy 300 + bare metal 2. deployment fails with errors Actual results: (undercloud) [stack@f17-h22-000-1029p ~]$ openstack stack failures list overcloud overcloud.1029uCompute.46.StoragePort: resource_type: OS::TripleO::1029uCompute::Ports::StoragePort physical_resource_id: 8a612a7e-de81-41be-b11b-e35edfa6ca81 status: UPDATE_FAILED status_reason: | RemoteError: resources.StoragePort: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029uCompute.47: resource_type: OS::TripleO::1029uCompute physical_resource_id: d12cf619-c5b7-4211-9d44-10b0c9a69a44 status: UPDATE_FAILED status_reason: | resources[47]: HEAT-E99001 Service nova is not available for resource type OS::TripleO::1029uComputeServer, reason: Could not determine a suitable URL for the plugin overcloud.1029uCompute.6.RoleUserData: resource_type: OS::TripleO::1029uCompute::NodeUserData physical_resource_id: a811b419-b177-4db3-b255-2da296b30606 status: UPDATE_FAILED status_reason: | RemoteError: resources.RoleUserData: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029uCompute.30.UpdateConfig: resource_type: OS::TripleO::Tasks::PackageUpdate physical_resource_id: 2f3cb9d0-6e8e-440a-98b5-703eefee41a3 status: UPDATE_FAILED status_reason: | RemoteError: resources.UpdateConfig: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029uCompute.13.TenantPort: resource_type: OS::TripleO::1029uCompute::Ports::TenantPort physical_resource_id: 70116e67-084e-4379-b449-ecf12fad03d2 status: UPDATE_FAILED status_reason: | RemoteError: resources.TenantPort: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029uCompute.17.NodeUserData: resource_type: OS::TripleO::NodeUserData physical_resource_id: 947b14c8-7cb2-4729-82d7-be4bf9f7e11b status: UPDATE_FAILED status_reason: | RemoteError: resources.NodeUserData: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029uCompute.19: resource_type: OS::TripleO::1029uCompute physical_resource_id: 3ea8b490-2e95-4451-a81e-d3b475cba066 status: UPDATE_FAILED status_reason: | resources[19]: HEAT-E99001 Service nova is not available for resource type OS::TripleO::1029uComputeServer, reason: Could not determine a suitable URL for the plugin overcloud.1029pCompute.52.StoragePort: resource_type: OS::TripleO::1029pCompute::Ports::StoragePort physical_resource_id: 3df6ca33-c279-4b82-aa36-9af724feffdf status: UPDATE_FAILED status_reason: | RemoteError: resources.StoragePort: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029pCompute.20: resource_type: OS::TripleO::1029pCompute physical_resource_id: 09581c65-edd8-4271-9e5f-ae455e3aeeff status: UPDATE_FAILED status_reason: | RemoteError: resources[20]: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029pCompute.38.UpdateConfig: resource_type: OS::TripleO::Tasks::PackageUpdate physical_resource_id: 00ef1523-ddb4-4a64-8bc4-90a1d5ee3e0f status: UPDATE_FAILED status_reason: | RemoteError: resources.UpdateConfig: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.1029pCompute.16.InternalApiPort: resource_type: OS::TripleO::1029pCompute::Ports::InternalApiPort physical_resource_id: 32fb01ba-47c8-4fcb-b696-9e2373603ba2 status: UPDATE_FAILED status_reason: | resources.InternalApiPort: HEAT-E99001 Service neutron is not available for resource type OS::Neutron::Port, reason: Could not determine a suitable URL for the plugin overcloud.1029pCompute.16.StoragePort: resource_type: OS::TripleO::1029pCompute::Ports::StoragePort physical_resource_id: 73c44a74-58c2-4ddf-9b6a-8368fd2ca66b status: UPDATE_FAILED status_reason: | resources.StoragePort: HEAT-E99001 Service neutron is not available for resource type OS::Neutron::Port, reason: Could not determine a suitable URL for the plugin overcloud.5039msCompute.69: resource_type: OS::TripleO::5039msCompute physical_resource_id: 52f24f96-beaf-4d1b-bd76-7b2dc85a84b1 status: UPDATE_FAILED status_reason: | RemoteError: resources[69]: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.5039msCompute.176: resource_type: OS::TripleO::5039msCompute physical_resource_id: e8674cd6-3e60-4b89-bc1f-76619d1d022f status: UPDATE_FAILED status_reason: | resources[176]: HEAT-E99001 Service nova is not available for resource type OS::TripleO::5039msComputeServer, reason: Could not determine a suitable URL for the plugin overcloud.5039msCompute.11.NodeUserData: resource_type: OS::TripleO::NodeUserData physical_resource_id: 5fe7d454-9d78-42de-8d17-f7949cc77624 status: UPDATE_FAILED status_reason: | RemoteError: resources.NodeUserData: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.r620Compute.54.TenantPort: resource_type: OS::TripleO::r620Compute::Ports::TenantPort physical_resource_id: b51aad3b-3f55-4d0e-beec-e62a358b967a status: UPDATE_FAILED status_reason: | RemoteError: resources.TenantPort: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.r620Compute.68: resource_type: OS::TripleO::r620Compute physical_resource_id: 5c8fcb6a-cb3c-4de5-9817-a79048cdfd05 status: UPDATE_FAILED status_reason: | RemoteError: resources[68]: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.r620Compute.0.TenantPort: resource_type: OS::TripleO::r620Compute::Ports::TenantPort physical_resource_id: 49b7f3c1-6624-4385-b412-05033d5ce4c1 status: UPDATE_FAILED status_reason: | RemoteError: resources.TenantPort: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.r620Compute.3.RoleUserData: resource_type: OS::TripleO::r620Compute::NodeUserData physical_resource_id: f4c8dac8-f384-4b24-b3d0-0e6fffa06f65 status: UPDATE_FAILED status_reason: | RemoteError: resources.RoleUserData: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.r620Compute.39.NetworkConfig: resource_type: OS::TripleO::r620Compute::Net::SoftwareConfig physical_resource_id: 998d842a-9ef9-4fba-b863-159f4fb26c81 status: UPDATE_FAILED status_reason: | RemoteError: resources.NetworkConfig: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. overcloud.r620Compute.12.UpdateConfig: resource_type: OS::TripleO::Tasks::PackageUpdate physical_resource_id: 6d4de743-2473-4ffa-acbc-fdc05978e473 status: UPDATE_FAILED status_reason: | RemoteError: resources.UpdateConfig: Remote error: DiscoveryFailure Could not determine a suitable URL for the plugin [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper\n result = f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 998, in update_stack\n is_registered_policy=True)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 182, in enforce_stack\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 163, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 148, in _enforce\n target=target, is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 106, in enforce\n is_registered_policy=is_registered_policy)\n', u' File "/usr/lib/python2.7/site-packages/heat/common/policy.py", line 77, in _check\n credentials = context.to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 208, in to_policy_values\n policy = super(RequestContext, self).to_policy_values()\n', u' File "/usr/lib/python2.7/site-packages/oslo_context/context.py", line 314, in to_policy_values\n \'roles\': self.roles,\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 301, in roles\n self._load_keystone_data()\n', u' File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 292, in _load_keystone_data\n auth_ref = self.auth_plugin.get_access(self.keystone_session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n', u' File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin\n raise exceptions.DiscoveryFailure(\'Could not determine a suitable URL \'\n', u'DiscoveryFailure: Could not determine a suitable URL for the plugin\n']. (undercloud) [stack@f17-h22-000-1029p ~]$
To provide more context, This happens during scale up from 252 nodes to 367 nodes, after all the new nodes go into ACTIVE. It happens consistently even on trying multiple redeploys. Keystone and heat both have memcached caching enabled. There are no errors in keystone logs. There are some shutdown_errors in the rabbitmq logs on undercloud. However, on retrying a stack update, we do not see rabbitmq errors but stack update still fails with the same error. Could this be possibly be related to rabbitmq going down during initial scale up and then stack going to a bad state, so that even subsequent updates to the stack are failing?
In heat logs.. Keystone requests are timing out... It's possibly keystone performance issue. I see 12 admin/public workers configured in keystone. May be those need to be increased if caching is not helping. Keystone team may be able to help here. 2019-08-12 03:37:25.874 284316 WARNING keystoneauth.identity.generic.base [req-a4e090be-c3e0-4e25-9c2c-1a2c7fd19a24 - admin - default default] Failed to discover available identity versions when conta cting http://192.168.0.1:35357. Attempting to parse version from URL.: ConnectFailure: Unable to establish connection to http://192.168.0.1:35357: HTTPConnectionPool(host='192.168.0.1', port=35357): M ax retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fcf87ba4c10>: Failed to establish a new connection: [Errno 110] ETIMEDO UT',))
Changing Keystone admin and main services to have 24 processes and 1 thread fixed the issue.
I don't think it's a good idea to increase the default keystone workers for undercloud. It uses ::os_workers fact whose value is max between '(<# processors> / 2)' and '2' with a cap of 12 for performance reasons[1]. We can possibly add it to the scale documentation among other things. If we need to keep the bug open for fixing docs, feel free to re-open it and assign it to the correct DFG. [1] https://github.com/openstack/puppet-openstacklib/commit/e6b658bd007924f0f71d0f2d8a6928976567cc98
Even after changes to keystone worker count, all further updates of the stack are failing. The errors in heat logs are as follows: https://gist.githubusercontent.com/smalleni/96ab99d19cc4a5b14ae0f79bfec6bbcb/raw/d0d04be7d166aaa50fe60012c0b24964eb6d2842/gistfile1.txt https://gist.githubusercontent.com/smalleni/d5bcb9ba419eaf573188507d317a531b/raw/4a2ff44c8ba2460af40b333d7574eb0a20bc8299/gistfile1.txt
We have tried with executor_thread_pool_size set to 32 as well as 16. We have even tried reducing heat engine workers to 12 and seeing if that fixes the problem but with 12 workers, the stack update does not progress. We have rpc_response_timeout set to 1200 in heat.conf.
Tweaks tried so far and current settings Heat Engine workers = 12/24 (tried 12 and 24, 24 is default) executor_thread_pool_size = 12/32/64 (64 is deault, tried other values as well) rpc_response_timeout (increased to 1200 from default 600) Keystone Increased admin to 48 processes and 1 thread Increased main to 24 processes and 1 thread Ironic sync_power_state_interval = 180 Mistral rpc_response_timeout=600
Hey Rabi, Many thanks for your keen interest to help us get past the issues. If I understand correctly, because of the keystoneauth change you applied manually, we would only need to apply 1. https://code.engineering.redhat.com/gerrit/#/c/178588/ 2. https://review.opendev.org/#/c/676733/ The following two patches are not needed because you manually patched /usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py ....... _logger.debug('Making authentication request to %s', token_url) resp = session.post(token_url, json=body, headers=headers, authenticated=False, log=False, connect_retries=10, **rkwargs) ....... 1. https://review.opendev.org/#/c/676664/ 2. https://review.opendev.org/#/c/676648/1 Is this correct?
Need to backport https://review.opendev.org/#/c/676821/ as well.
Post 400+ nodes, we are running with 48 workers and 48 threads for heat-engine. Another quick note, we are seeing mistral-engine bottlenecked on CPU as soon as we kick off a stack update at the 300+ node scale, as it creates/updates plan, which is slowing down the stack update. We need to figure out if we can run more than 1 worker for mistral-engine.
At around close to 400 nodes or a little less, another change that was made by Rabi to prevent too many large messages pushed to the queue when updating SshKnownHostsDeployment resources, that are replaced (deleted/created) for every node added to a role and a new config (with the new node details added) created. overcloud.j2.yaml ------------------------ {{role.name}}SshKnownHostsDeployment: type: OS::TripleO::Ssh::KnownHostsDeployment + update_policy: + rolling_update: + max_batch_size: {get_param: NodeCreateBatchSize} <<30 is the default for this parameter properties: name: {{role.name}}SshKnownHostsDeployment config: {get_resource: SshKnownHostsConfig} servers: {get_attr: [{{role.name}}Servers, value]} max_batch_size is 30 per role. We are not sure if this change was really needed but documenting for future reference.
> another change that was made by Rabi to prevent too many large messages pushed to the queue when updating SshKnownHostsDeployment resources I've reverted it after some testing today and it did not seem to make any difference.
To add to Rabi's observation. With the change overcloud.j2.yaml ------------------------ {{role.name}}SshKnownHostsDeployment: type: OS::TripleO::Ssh::KnownHostsDeployment + update_policy: + rolling_update: + max_batch_size: {get_param: NodeCreateBatchSize} <<30 is the default for this parameter properties: name: {{role.name}}SshKnownHostsDeployment config: {get_resource: SshKnownHostsConfig} servers: {get_attr: [{{role.name}}Servers, value]} Adding a node to a an existing 505 node overcloud takes 80 minutes. Without this change it takes 65 minutes. Given that this slows down the deployment and we are not currently seeing any rabbitmq issues without this patch, this does not seem necessary.
According to our records, this should be resolved by openstack-heat-10.0.3-8.el7ost. This build is available now.
Went through all patches and verified their existence
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0761