Bug 1419348

Summary:	Backport to OSP 8: OpenStack Heat may fail to connect keystone admin API in multi-region environment
Product:	Red Hat OpenStack	Reporter:	Andreas Karis <akaris>
Component:	openstack-heat	Assignee:	Rabi Mishra <ramishra>
Status:	CLOSED ERRATA	QA Contact:	Amit Ugol <augol>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	8.0 (Liberty)	CC:	augol, a-yoshiyama, dcadzow, mburns, ramishra, rhel-osp-director-maint, sbaker, sclewis, shardy, srevivo, zbitter
Target Milestone:	---	Keywords:	FeatureBackport, Triaged, ZStream
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-heat-5.0.3-2.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1418159	Environment:
Last Closed:	2017-06-14 15:26:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1418159, 1428632
Bug Blocks:

Description Andreas Karis 2017-02-05 17:11:54 UTC

+++ This bug was initially created as a clone of Bug #1418159 +++

Description of problem:
Heat uses Keystone Trust to build a stack as a tenant user. To do so, heat-engine needs connections to a Keystone admin API endpoint.
In multi-region environment, each region may have its own Keystone admin API and Keystone may have multiple endpoint entries. In this case, heat-engine uses the 1st. So, heat-engine in regions without the 1st endpoint tries to connect another region with the 1st. It will fail if there is no route to the endpoint.

Version-Release number of selected component (if applicable):
openstack-heat-common-6.0.0-8.el7ost.noarch

How reproducible:
100% in regions without the 1st endpoint.

Steps to Reproduce:
1. Register multiple Keystone admin endpoints. The 1st entry should have dummy region name and inaccessible IP address from heat-engine.
2. "heat stack-create <template>"
3.

Actual results:
heat command failed with and error:
ConnectFailure: Unable to establish connection to http://<inaccessbile>:35357/v2.0/OS-TRUST/trusts

Expected results:
heat-engine works properly and the stack creation completed.

Additional info:
The attached patch fixes the problem.

--- Additional comment from Zane Bitter on 2017-02-02 10:57:54 EST ---

Can you please submit this patch upstream and post a link to the review here? I will make sure it gets reviewed and backported. Thanks!

--- Additional comment from Rabi Mishra on 2017-02-03 08:02:00 EST ---

I've raised an upstream bug and put a fix for it.

Comment 1 Andreas Karis 2017-02-05 17:13:16 UTC

Once the upstream bug and the bugfix of OSP 10/9 are through, I'd like to request a backport to OSP 8. A customer successfully tested this on OSP 8, so it shouldn't be too complicated to backport:

~~~
--- ./heat/common/heat_keystoneclient.py.orig   2016-08-25 18:21:42.000000000 +0000
+++ ./heat/common/heat_keystoneclient.py        2017-02-05 00:23:08.730154392 +0000
@@ -112,6 +112,10 @@
         return self._client

     @property
+    def region_name(self):
+        return self.context.region_name or cfg.CONF.region_name_for_services
+
+    @property
     def domain_admin_auth(self):
         if not self._domain_admin_auth:
             # Note we must specify the domain when getting the token
@@ -140,13 +144,15 @@
         if not self._domain_admin_client:
             self._domain_admin_client = kc_v3.Client(
                 session=self.session,
-                auth=self.domain_admin_auth)
+                auth=self.domain_admin_auth,
+                region_name=self.region_name)

         return self._domain_admin_client

     def _v3_client_init(self):
         client = kc_v3.Client(session=self.session,
-                              auth=self.context.auth_plugin)
+                              auth=self.context.auth_plugin,
+                              region_name=self.region_name)

         if hasattr(self.context.auth_plugin, 'get_access'):
             # NOTE(jamielennox): get_access returns the current token without
~~~

Comment 7 errata-xmlrpc 2017-06-14 15:26:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1456