Bug 1639184

I see this same thing occurring on an upstream queens TripleO deployment.  In my case, it doesn't seem specific to the EC2 API (a 'openstack user list' shows the same behavior).

There is an interesting clue in the verbose output from the 'openstack' CLI.  Consider the following example:

----------------------------------------------------------------------------------------------
Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
Making authentication request to https://overcloud.ooo.test:13000/v3/auth/tokens
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "ebf38ae4fb034a55bdb7c8e941fce39d", "name": "admin"}], "expires_at": "2018-11-29T22:03:22.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "18faf0f2cbcb4eee8577529ad7328712", "name": "admin"}, "catalog": [...], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "db7aa21c695542ffa3907a8020d3ee71"}, "audit_ids": ["7IeZwGzqRpWI-tYEK1HjPg"], "issued_at": "2018-11-29T21:03:22.000000Z"}}
----------------------------------------------------------------------------------------------

The client successfully obtained a token from Keystone here using my public URL.  This is followed immediately by a version discovery request against keystone, but it uses the admin endpoint as defined in the catalog that was returned as a part of the above token:


----------------------------------------------------------------------------------------------
REQ: curl -g -i -X GET https://overcloud.ctlplane.ooo.test:35357 -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"
RESP: [300] Date: Thu, 29 Nov 2018 21:03:22 GMT Server: Apache Vary: X-Auth-Token Content-Length: 633 Content-Type: application/json 
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "https://overcloud.ctlplane.ooo.test:35357/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://overcloud.ctlplane.ooo.test:35357/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}
----------------------------------------------------------------------------------------------

Since the client accessed keystone using the admin endpoint URL, the returned links for the v2.0 and v3 APIs also use the admin endpoint.  I am still trying to track down where the version discovery call is coming from so we can see why it is trying to use the admin URL.  This is likely in python-keystoneclient or in python-openstackclient.

The discovered URLs are then used for the actual call to list EC2 tokens (or whatever other API you are tryign to use):

----------------------------------------------------------------------------------------------
REQ: curl -g -i -X GET https://overcloud.ctlplane.ooo.test:35357/v3/users/db7aa21c695542ffa3907a8020d3ee71/credentials/OS-EC2 -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}...
----------------------------------------------------------------------------------------------

This issue seems to have a simple workaround (at least in OSP13 where I tested it).  It seems that python-keystoneclient attempts to use the "admin" interface by default.  It is not clear if this is considered to be a bug.  There is an option that can be set to tell it what interface to use.  With the 'openstack' CLI, you can use --os-interface like so:

  openstack --os-interface=public -vv ec2 credentials list

You can also set the OS_INTERFACE environment variable instead of using the command line option.  You will see that the public URL is used for all calls in the verbose output.

Will this satisfy the customer?

The default interface choice of 'admin' can be seen here in python-keystoneclient:

    https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/httpclient.py#L223-L224

It seems like the expected way of handling this is to use the options mentioned in my previous comment.

A patch has just been merged into the master branch (which will become the stein release) of osc-lib upstream that changes the default for --os-interface to "public".  This library is used by python-openstack client for the 'openstack' CLI:

   https://review.openstack.org/#/c/621276

I do not think it is appropriate to backport this change to earlier stable branches, as the issue has an easy workaround of specifying --os-interface=public or setting OS_INTERFACE=public.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811