+++ This bug was initially created as a clone of Bug #1639183 +++ Description of problem: Keystone v2.0 provides different set of URLs for different kinds of requests: - adminurl: for admin requests - internalurl: for internal services - publicurl: for public requests Customer reported the following problem: openstackclient requires access to adminURL in cases when it is not needed. For example, every user should be able to create and list his EC2 credentials (""openstack ec2 credentials list"" command), but for some reason openstack client uses adminURL to process such requests and hangs if it is not available. Version-Release number of selected component (if applicable): I was able to reproduce this issue for RHOSP 10, RHOSP 11 and RHOSP 12 (keystone v2.0 was deprecated in Queens, so haven't checked it). How reproducible: Collect information about identity endpoint, get its URLs. Use EC2 commands with active verbose flags to create and list EC2 credentials: openstack -vv ec2 credentials create openstack -vv ec2 credentials list You will see that it pools adminURL (generally uses 35357 port). You could also block this URL with iptables and see that request will timeout. Expected results: User will be able to get his EC2 credentials without polling internal URL. Additional info: Though I report this issue for certain command, it seems that other commands are also affected and should be investigated. Workaround: Create DNAT rule that will DNAT adminURL to publicURL.
I see this same thing occurring on an upstream queens TripleO deployment. In my case, it doesn't seem specific to the EC2 API (a 'openstack user list' shows the same behavior). There is an interesting clue in the verbose output from the 'openstack' CLI. Consider the following example: ---------------------------------------------------------------------------------------------- Instantiating identity client: <class 'keystoneclient.v3.client.Client'> Making authentication request to https://overcloud.ooo.test:13000/v3/auth/tokens {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "ebf38ae4fb034a55bdb7c8e941fce39d", "name": "admin"}], "expires_at": "2018-11-29T22:03:22.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "18faf0f2cbcb4eee8577529ad7328712", "name": "admin"}, "catalog": [...], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "db7aa21c695542ffa3907a8020d3ee71"}, "audit_ids": ["7IeZwGzqRpWI-tYEK1HjPg"], "issued_at": "2018-11-29T21:03:22.000000Z"}} ---------------------------------------------------------------------------------------------- The client successfully obtained a token from Keystone here using my public URL. This is followed immediately by a version discovery request against keystone, but it uses the admin endpoint as defined in the catalog that was returned as a part of the above token: ---------------------------------------------------------------------------------------------- REQ: curl -g -i -X GET https://overcloud.ctlplane.ooo.test:35357 -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" RESP: [300] Date: Thu, 29 Nov 2018 21:03:22 GMT Server: Apache Vary: X-Auth-Token Content-Length: 633 Content-Type: application/json RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "https://overcloud.ctlplane.ooo.test:35357/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://overcloud.ctlplane.ooo.test:35357/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}} ---------------------------------------------------------------------------------------------- Since the client accessed keystone using the admin endpoint URL, the returned links for the v2.0 and v3 APIs also use the admin endpoint. I am still trying to track down where the version discovery call is coming from so we can see why it is trying to use the admin URL. This is likely in python-keystoneclient or in python-openstackclient. The discovered URLs are then used for the actual call to list EC2 tokens (or whatever other API you are tryign to use): ---------------------------------------------------------------------------------------------- REQ: curl -g -i -X GET https://overcloud.ctlplane.ooo.test:35357/v3/users/db7aa21c695542ffa3907a8020d3ee71/credentials/OS-EC2 -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}... ----------------------------------------------------------------------------------------------
This issue seems to have a simple workaround (at least in OSP13 where I tested it). It seems that python-keystoneclient attempts to use the "admin" interface by default. It is not clear if this is considered to be a bug. There is an option that can be set to tell it what interface to use. With the 'openstack' CLI, you can use --os-interface like so: openstack --os-interface=public -vv ec2 credentials list You can also set the OS_INTERFACE environment variable instead of using the command line option. You will see that the public URL is used for all calls in the verbose output. Will this satisfy the customer?
The default interface choice of 'admin' can be seen here in python-keystoneclient: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/httpclient.py#L223-L224 It seems like the expected way of handling this is to use the options mentioned in my previous comment.
A patch has just been merged into the master branch (which will become the stein release) of osc-lib upstream that changes the default for --os-interface to "public". This library is used by python-openstack client for the 'openstack' CLI: https://review.openstack.org/#/c/621276 I do not think it is appropriate to backport this change to earlier stable branches, as the issue has an easy workaround of specifying --os-interface=public or setting OS_INTERFACE=public.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811