Description of problem: When keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest* are executed in a public_endpoint=https://example.com and admin_endpoint=http://<admin_endpoint_URL>, the tests fail like below. ~~~ {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_authorize_request_token [1.248035s] ... FAILED {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_create_access_token [0.504674s] ... FAILED {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_create_request_token [0.736799s] ... FAILED {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_list_access_tokens [1.053999s] ... FAILED {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_list_roles_for_access_token [0.756529s] ... FAILED {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_revoke_access_token [0.772879s] ... FAILED {0} keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_show_role_for_access_token [0.557021s] ... FAILED ~~~ Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Make public_endpoint and admin_endpoint URLs scheme different. https:// vs https:// 2. Run keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest.test_authorize_request_token test, for example. Actual results: The test fail. Expected results: No tests fail. The public_endpoint option should be used for only public_endpoint. Additional info: I found that the sig base string contains "https" even though the request is against the admin URL which doesn't contain "https" but "http" like below. ~~~ 2020-07-16 13:22:46.321 262 DEBUG keystone.server.flask.request_processing.req_logging [] SCRIPT_NAME: `` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:28 2020-07-16 13:22:46.321 262 DEBUG keystone.server.flask.request_processing.req_logging [] PATH_INFO: `/v3/OS-OAUTH1/request_token` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29 2020-07-16 13:22:46.332 262 DEBUG oauthlib.oauth1.rfc5849.signature [] Verify HMAC-SHA1 failed: sig base string: POST&https%3A%2F%2F<IPADDRESS1>%3A35357%2Fv3%2FOS-OAUTH1%2Frequest_token&oauth_callback%3Doob%26oauth_consumer_key%3D4deadbeafdbeafdeadbeafdeadbeaf%26oauth_nonce%3D1999999999999999999999999%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1594873366%26oauth_version%3D1.0 verify_hmac_sha1 /usr/lib/python3.6/site-packages/oauthlib/oauth1/rfc5849/signature.py:576 2020-07-16 13:22:46.332 262 INFO oauthlib.oauth1.rfc5849.endpoints.request_token [] [Failure] request verification failed. 2020-07-16 13:22:46.332 262 INFO oauthlib.oauth1.rfc5849.endpoints.request_token [] Valid client: True. 2020-07-16 13:22:46.333 262 INFO oauthlib.oauth1.rfc5849.endpoints.request_token [] Valid realm: True. 2020-07-16 13:22:46.333 262 INFO oauthlib.oauth1.rfc5849.endpoints.request_token [] Valid callback: True. 2020-07-16 13:22:46.333 262 INFO oauthlib.oauth1.rfc5849.endpoints.request_token [] Valid signature: False. 2020-07-16 13:22:46.336 262 WARNING keystone.server.flask.application [] Authorization failed. The request you have made requires authentication. from <IPADDRESS-2>: keystone.exception.Unauthorized: The request you have made requires authentication. ~~~ (overcloud) $ openstack endpoint list ... | <ID1> | regionOne | keystone | identity | True | public | https://openstack.PUBLICENDPOINT:13000 | | <ID2> | regionOne | keystone | identity | True | admin | http://<IPADDRESS1>:35357 | | <ID3> | regionOne | keystone | identity | True | internal | http://<IPADDRESS1>:5000 | ... /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf: [DEFAULT] ... public_endpoint=https://openstack.PUBLICENDPOINT:13000 ~~~ It seems this is caused by base_url comes from this code. https://github.com/openstack/keystone/blob/stable/train/keystone/server/flask/common.py#L1005
Hi Masayuki, Are you updating the public endpoint or the admin endpoint (the schemes referenced in the description are identical)? Keystone uses a separate library to generate signatures with OAUTH1 parameters and it always passes in the public endpoint [0 - 2]. The error is likely from the URL scheme mismatching the request URL and what's configured in keystone [3]. Based on the code, it appears OAUTH1 isn't designed to work with admin endpoints. [0] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/api/os_oauth1.py#L158 [1] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/api/os_oauth1.py#L63-L70 [2] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/server/flask/common.py#L1005 [3] https://docs.openstack.org/keystone/latest/configuration/config-options.html#DEFAULT.public_endpoint
Hi Lance, Thank you for your reply! Yes, the customer updated the public endpoint and the admin endpoint like in the description. So, the scheme of public endpoint is "https" and admin's one is "http". public endpoint: https://openstack.PUBLICENDPOINT:13000 admin endpoint : http://<IPADDRESS1>:35357 In tempest side, the OAUTH1 signature is created in oauth_token_client.py[1], and the uri comes from auth.py. Its endpoint_type is "adminURL" by default from the code[3]. So, in this case, are we supposed to change the config - v3_endpoint_type to 'publicURL'(or just 'public') in tempest.conf? And, I'm wondering whether any other inconsistency happens if OAUTH1 doesn't work with admin endpoints. I have no concrete example yet, though. [1] https://opendev.org/openstack/tempest/src/branch/master/tempest/lib/services/identity/v3/oauth_token_client.py#L42-L98 [2] https://opendev.org/openstack/tempest/src/branch/master/tempest/lib/auth.py#L503-L588 [3] https://opendev.org/openstack/tempest/src/branch/master/tempest/config.py#L140-L147
Hi Lance, IIUC the root cause of this problem is that base_url method[1][2], which is generally used to detect endpoint url in keystone, depends on a single "public_endpoint" parameter, while in TripleO deployment each keystone endpoint(admin/internal/public) has different urls. [1] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/server/flask/common.py#L1005 [2] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/server/flask/request_processing/middleware/auth_context.py#L75 As you mentioned we can avoid the api erorrs if we force oauth request to go to public endpoint, but I don't think that's ideal solution here because keystone still "accepts" oauth request in admin endpoint. In addition, the problem was identified with oauth1 so far, but I'm afraid this behavior can potentially cause incorrect url in other APIs, when a request arrives at admin/internal api, regarding the fact that base_url method is widely used in keystone. As far as I've checked the current implementation in keystone, it seems that keystone doesn't request public_endpoint parameter set, but it can detect urls based on http headers passed from haproxy[3]. [3] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/server/flask/common.py#L1013 We suggested the customer to remove public_endpoint from keystone.conf and they confirmed removing that parameter solves the tempest failure. Based on that confirmation, I have filed a bug on TripleO [4] and submitted some patches to unset public_endpoint parameter, to make sure that keystone depends on these header values instead of the single parameter in keystone.conf. [4] https://bugs.launchpad.net/tripleo/+bug/1889017 May I ask your opinion about these fixes ? In addition, honestly speaking current endpoint detection with public_endpoint seems improper, because it assumes that all keystone endpoints should use a single host name, and IMO keystone should provide some mechanism to allow overriding each endpoints separately. It would be nice if I can ask you opinion about this as well. Thank you Takashi
Hi Takashi, I think you're on the right track with removing the public_endpoint configuration option from keystone. I did some testing locally and I think deriving the base_url from the request environment is the right thing to do [0]. As you noted, public_endpoint must be None for this to happen. When I remove public_endpoint from keystone's configuration file, keystone appears to build requests appropriately, regardless of the endpoint used to make the request: [heat-admin@overcloud-controller-0 ~]$ sudo head -25 /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf [DEFAULT] # # From keystone # <snip> # The base public endpoint URL for Keystone that is advertised to clients # (NOTE: this does NOT affect how Keystone listens for connections). Defaults # to the base host URL of the request. For example, if keystone receives a # request to `http://server:5000/v3/users`, then this will option will be # automatically treated as `http://server:5000`. You should only need to set # option if either the value of the base URL contains a path that keystone does # not automatically infer (`/prefix/v3`), or if the endpoint should be found on # a different host. (uri value) #public_endpoint = <None> #public_endpoint=https://overcloud.ooo.test:13000 [stack@undercloud ~]$ openstack endpoint list --os-cloud overcloud --service identity +----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------+ | 1d1fa7097eca427886c604caf4b940b6 | regionOne | keystone | identity | True | public | https://overcloud.ooo.test:13000 | | 7944a8d7e2834c828ee9120e29c69853 | regionOne | keystone | identity | True | admin | https://overcloud.ctlplane.ooo.test:35357 | | f1b1edbe814e4fb393246317d1293ff4 | regionOne | keystone | identity | True | internal | https://overcloud.internalapi.ooo.test:5000 | +----------------------------------+-----------+--------------+--------------+---------+-----------+---------------------------------------------+ [stack@undercloud ~]$ curl -X GET -H "X-Auth-Token: $TOKEN" https://overcloud.ctlplane.ooo.test:35357/v3/projects | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 774 100 774 0 0 9797 0 --:--:-- --:--:-- --:--:-- 9797 { "projects": [ { "id": "bf7ce0b21a7841b69ab0919478e6d56e", "name": "admin", "domain_id": "default", "description": "Bootstrap project for initializing the cloud.", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.ctlplane.ooo.test:35357/v3/projects/bf7ce0b21a7841b69ab0919478e6d56e" } }, { "id": "ef589aea25304b728f39c587f8c472e4", "name": "service", "domain_id": "default", "description": null, "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.ctlplane.ooo.test:35357/v3/projects/ef589aea25304b728f39c587f8c472e4" } } ], "links": { "next": null, "self": "https://overcloud.ctlplane.ooo.test:35357/v3/projects", "previous": null } } [stack@undercloud ~]$ curl -X GET -H "X-Auth-Token: $TOKEN" https://overcloud.internalapi.ooo.test:5000/v3/projects | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 780 100 780 0 0 8478 0 --:--:-- --:--:-- --:--:-- 8478 { "projects": [ { "id": "bf7ce0b21a7841b69ab0919478e6d56e", "name": "admin", "domain_id": "default", "description": "Bootstrap project for initializing the cloud.", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.internalapi.ooo.test:5000/v3/projects/bf7ce0b21a7841b69ab0919478e6d56e" } }, { "id": "ef589aea25304b728f39c587f8c472e4", "name": "service", "domain_id": "default", "description": null, "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.internalapi.ooo.test:5000/v3/projects/ef589aea25304b728f39c587f8c472e4" } } ], "links": { "next": null, "self": "https://overcloud.internalapi.ooo.test:5000/v3/projects", "previous": null } } [stack@undercloud ~]$ curl -X GET -H "X-Auth-Token: $TOKEN" https://overcloud.ooo.test:13000/v3/projects | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 747 100 747 0 0 7701 0 --:--:-- --:--:-- --:--:-- 7701 { "projects": [ { "id": "bf7ce0b21a7841b69ab0919478e6d56e", "name": "admin", "domain_id": "default", "description": "Bootstrap project for initializing the cloud.", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.ooo.test:13000/v3/projects/bf7ce0b21a7841b69ab0919478e6d56e" } }, { "id": "ef589aea25304b728f39c587f8c472e4", "name": "service", "domain_id": "default", "description": null, "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.ooo.test:13000/v3/projects/ef589aea25304b728f39c587f8c472e4" } } ], "links": { "next": null, "self": "https://overcloud.ooo.test:13000/v3/projects", "previous": null } } The self links are built accorindg to the endpoint in the request. Using TripleO public_endpoint default causes a mismatch: [heat-admin@overcloud-controller-0 ~]$ sudo head -25 /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf [DEFAULT] # # From keystone # # The base public endpoint URL for Keystone that is advertised to clients # (NOTE: this does NOT affect how Keystone listens for connections). Defaults # to the base host URL of the request. For example, if keystone receives a # request to `http://server:5000/v3/users`, then this will option will be # automatically treated as `http://server:5000`. You should only need to set # option if either the value of the base URL contains a path that keystone does # not automatically infer (`/prefix/v3`), or if the endpoint should be found on # a different host. (uri value) #public_endpoint = <None> public_endpoint=https://overcloud.ooo.test:13000 [stack@undercloud ~]$ curl -X GET -H "X-Auth-Token: $TOKEN" https://overcloud.ctlplane.ooo.test:35357/v3/projects | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 747 100 747 0 0 1765 0 --:--:-- --:--:-- --:--:-- 1761 { "projects": [ { "id": "bf7ce0b21a7841b69ab0919478e6d56e", "name": "admin", "domain_id": "default", "description": "Bootstrap project for initializing the cloud.", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.ooo.test:13000/v3/projects/bf7ce0b21a7841b69ab0919478e6d56e" } }, { "id": "ef589aea25304b728f39c587f8c472e4", "name": "service", "domain_id": "default", "description": null, "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": { "self": "https://overcloud.ooo.test:13000/v3/projects/ef589aea25304b728f39c587f8c472e4" } } ], "links": { "next": null, "self": "https://overcloud.ooo.test:13000/v3/projects", "previous": null } } I think removing public_endpoint from keystone's configuration is the right path forward unless TripleO folks have other reasons to keep it around. [0] https://opendev.org/openstack/keystone/src/branch/stable/train/keystone/server/flask/common.py#L1012-L1019
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284