Bug 1419348 - Backport to OSP 8: OpenStack Heat may fail to connect keystone admin API in multi-region environment
Summary: Backport to OSP 8: OpenStack Heat may fail to connect keystone admin API in m...
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Rabi Mishra
QA Contact: Amit Ugol
Depends On: 1418159 1428632
TreeView+ depends on / blocked
Reported: 2017-02-05 17:11 UTC by Andreas Karis
Modified: 2020-03-11 15:44 UTC (History)
11 users (show)

Fixed In Version: openstack-heat-5.0.3-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1418159
Last Closed: 2017-06-14 15:26:17 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Launchpad 1661594 0 None None None 2017-02-05 17:11:54 UTC
OpenStack gerrit 428718 0 None None None 2017-02-05 17:11:54 UTC
Red Hat Product Errata RHSA-2017:1456 0 normal SHIPPED_LIVE Low: openstack-heat security and bug fix update 2017-06-14 19:18:02 UTC

Description Andreas Karis 2017-02-05 17:11:54 UTC
+++ This bug was initially created as a clone of Bug #1418159 +++

Description of problem:
Heat uses Keystone Trust to build a stack as a tenant user. To do so, heat-engine needs connections to a Keystone admin API endpoint.
In multi-region environment, each region may have its own Keystone admin API and Keystone may have multiple endpoint entries. In this case, heat-engine uses the 1st. So, heat-engine in regions without the 1st endpoint tries to connect another region with the 1st. It will fail if there is no route to the endpoint.

Version-Release number of selected component (if applicable):

How reproducible:
100% in regions without the 1st endpoint.

Steps to Reproduce:
1. Register multiple Keystone admin endpoints. The 1st entry should have dummy region name and inaccessible IP address from heat-engine.
2. "heat stack-create <template>"

Actual results:
heat command failed with and error:
ConnectFailure: Unable to establish connection to http://<inaccessbile>:35357/v2.0/OS-TRUST/trusts

Expected results:
heat-engine works properly and the stack creation completed.

Additional info:
The attached patch fixes the problem.

--- Additional comment from Zane Bitter on 2017-02-02 10:57:54 EST ---

Can you please submit this patch upstream and post a link to the review here? I will make sure it gets reviewed and backported. Thanks!

--- Additional comment from Rabi Mishra on 2017-02-03 08:02:00 EST ---

I've raised an upstream bug and put a fix for it.

Comment 1 Andreas Karis 2017-02-05 17:13:16 UTC
Once the upstream bug and the bugfix of OSP 10/9 are through, I'd like to request a backport to OSP 8. A customer successfully tested this on OSP 8, so it shouldn't be too complicated to backport:

--- ./heat/common/heat_keystoneclient.py.orig   2016-08-25 18:21:42.000000000 +0000
+++ ./heat/common/heat_keystoneclient.py        2017-02-05 00:23:08.730154392 +0000
@@ -112,6 +112,10 @@
         return self._client

+    def region_name(self):
+        return self.context.region_name or cfg.CONF.region_name_for_services
+    @property
     def domain_admin_auth(self):
         if not self._domain_admin_auth:
             # Note we must specify the domain when getting the token
@@ -140,13 +144,15 @@
         if not self._domain_admin_client:
             self._domain_admin_client = kc_v3.Client(
-                auth=self.domain_admin_auth)
+                auth=self.domain_admin_auth,
+                region_name=self.region_name)

         return self._domain_admin_client

     def _v3_client_init(self):
         client = kc_v3.Client(session=self.session,
-                              auth=self.context.auth_plugin)
+                              auth=self.context.auth_plugin,
+                              region_name=self.region_name)

         if hasattr(self.context.auth_plugin, 'get_access'):
             # NOTE(jamielennox): get_access returns the current token without

Comment 7 errata-xmlrpc 2017-06-14 15:26:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.