Bug 1639184 - [RHOSP11]openstack client requires access to keystone's v2.0 adminURL, when it really shouldn't
Summary: [RHOSP11]openstack client requires access to keystone's v2.0 adminURL, when i...
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-osc-lib
Version: 15.0 (Stein)
Hardware: All
OS: Linux
Target Milestone: ga
: 15.0 (Stein)
Assignee: RHOS Maint
QA Contact: Pavan
Depends On: 1639183 1639185
TreeView+ depends on / blocked
Reported: 2018-10-15 09:02 UTC by Alex Stupnikov
Modified: 2019-10-22 07:20 UTC (History)
14 users (show)

Fixed In Version: python-osc-lib-1.12.1-0.20190510040402.9b610ea.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1639183
Last Closed: 2019-09-21 11:19:11 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
OpenStack gerrit 621276 0 None MERGED Default to using public interface for all clients 2020-03-17 15:40:26 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:19:30 UTC

Description Alex Stupnikov 2018-10-15 09:02:20 UTC
+++ This bug was initially created as a clone of Bug #1639183 +++

Description of problem:

Keystone v2.0 provides different set of URLs for different kinds of requests:

- adminurl: for admin requests
- internalurl: for internal services
- publicurl: for public requests

Customer reported the following problem: openstackclient requires access to adminURL in cases when it is not needed. For example, every user should be able to create and list his EC2 credentials (""openstack ec2 credentials list"" command), but for some reason openstack client uses adminURL to process such requests and hangs if it is not available.

Version-Release number of selected component (if applicable):

I was able to reproduce this issue for RHOSP 10, RHOSP 11 and RHOSP 12 (keystone v2.0 was deprecated in Queens, so haven't checked it).

How reproducible:

Collect information about identity endpoint, get its URLs.

Use EC2 commands with  active verbose flags to create and list EC2 credentials:

openstack -vv ec2 credentials create
openstack -vv ec2 credentials list

You will see that it pools adminURL (generally uses 35357 port). You could also block this URL with iptables and see that request will timeout.

Expected results:

User will be able to get his EC2 credentials without polling internal URL.

Additional info:

Though I report this issue for certain command, it seems that other commands are also affected and should be investigated.


Create DNAT rule that will DNAT adminURL to publicURL.

Comment 5 Nathan Kinder 2018-11-29 22:18:29 UTC
I see this same thing occurring on an upstream queens TripleO deployment.  In my case, it doesn't seem specific to the EC2 API (a 'openstack user list' shows the same behavior).

There is an interesting clue in the verbose output from the 'openstack' CLI.  Consider the following example:

Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
Making authentication request to https://overcloud.ooo.test:13000/v3/auth/tokens
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "ebf38ae4fb034a55bdb7c8e941fce39d", "name": "admin"}], "expires_at": "2018-11-29T22:03:22.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "18faf0f2cbcb4eee8577529ad7328712", "name": "admin"}, "catalog": [...], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "db7aa21c695542ffa3907a8020d3ee71"}, "audit_ids": ["7IeZwGzqRpWI-tYEK1HjPg"], "issued_at": "2018-11-29T21:03:22.000000Z"}}

The client successfully obtained a token from Keystone here using my public URL.  This is followed immediately by a version discovery request against keystone, but it uses the admin endpoint as defined in the catalog that was returned as a part of the above token:

REQ: curl -g -i -X GET https://overcloud.ctlplane.ooo.test:35357 -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"
RESP: [300] Date: Thu, 29 Nov 2018 21:03:22 GMT Server: Apache Vary: X-Auth-Token Content-Length: 633 Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "https://overcloud.ctlplane.ooo.test:35357/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://overcloud.ctlplane.ooo.test:35357/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Since the client accessed keystone using the admin endpoint URL, the returned links for the v2.0 and v3 APIs also use the admin endpoint.  I am still trying to track down where the version discovery call is coming from so we can see why it is trying to use the admin URL.  This is likely in python-keystoneclient or in python-openstackclient.

The discovered URLs are then used for the actual call to list EC2 tokens (or whatever other API you are tryign to use):

REQ: curl -g -i -X GET https://overcloud.ctlplane.ooo.test:35357/v3/users/db7aa21c695542ffa3907a8020d3ee71/credentials/OS-EC2 -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}...

Comment 6 Nathan Kinder 2018-11-29 22:41:07 UTC
This issue seems to have a simple workaround (at least in OSP13 where I tested it).  It seems that python-keystoneclient attempts to use the "admin" interface by default.  It is not clear if this is considered to be a bug.  There is an option that can be set to tell it what interface to use.  With the 'openstack' CLI, you can use --os-interface like so:

  openstack --os-interface=public -vv ec2 credentials list

You can also set the OS_INTERFACE environment variable instead of using the command line option.  You will see that the public URL is used for all calls in the verbose output.

Will this satisfy the customer?

Comment 7 Nathan Kinder 2018-11-29 23:11:35 UTC
The default interface choice of 'admin' can be seen here in python-keystoneclient:


It seems like the expected way of handling this is to use the options mentioned in my previous comment.

Comment 11 Nathan Kinder 2018-12-01 21:36:34 UTC
A patch has just been merged into the master branch (which will become the stein release) of osc-lib upstream that changes the default for --os-interface to "public".  This library is used by python-openstack client for the 'openstack' CLI:


I do not think it is appropriate to backport this change to earlier stable branches, as the issue has an easy workaround of specifying --os-interface=public or setting OS_INTERFACE=public.

Comment 19 errata-xmlrpc 2019-09-21 11:19:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.