Created attachment 1549132 [details] Updates octavia-base to use v3 endpoint for service_auth url Description of problem: After updating to the latest z5 release of RHOSP 13, trying to create a TERMINATED_HTTPS loadbalancer listener is failing with 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request. Version-Release number of selected component (if applicable): registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-66 python-octavia-2.0.3-2.el7ost.noarch puppet-octavia-12.4.0-8.el7ost.noarch openstack-octavia-common-2.0.3-2.el7ost.noarch openstack-octavia-api-2.0.3-2.el7ost.noarch openstack-tripleo-heat-templates-8.2.0-6.1.el7ost.noarch How reproducible: Easy to reproduce. Following the same series of steps against the scenario when registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-64 is the image used by octavia_api works, but fails with 13.0-66. Steps to Reproduce: Standard steps for creating a TERMINATED_HTTPS loadbalancer - openssl pkcs12 -export -inkey server.key -in server.crt -certfile ca-chain.crt -passout pass: -out server.p12 openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)" openstack acl user add -u <octavia_user_uuid> $(openstack secret list | awk '/ tls_secret1 / {print $2}') openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}') lb1 Actual results: Creation fails with: 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request Expected results: Loadbalancer listener is created Additional info: The new version of the Octavia image looks like it includes some enhancements that are designed to automatically add the octavia user uuid to the barbican secret containing the certificate data Presumably, this is related to this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1569129 From what I can tell, the issue with creating a TERMINATED_HTTPS listener against the new code is that the auth_url in the [service_auth] section of octavia.conf is being configured by tripleo to use KeystonePublic as the endpoint and not KeystoneV3Public. Hand-editing octavia.conf and adding "/v3" to the auth_url, restarting the service, and running through the 'steps to reproduce' results in a successful loadbalancer listener creation. The value for auth_url comes from the following line in /usr/share/openstack-tripleo-heat-templates/puppet/services/octavia-base.yaml: octavia::service_auth::auth_url: {get_param: [EndpointMap, KeystonePublic, uri]} I've attached a patch that simply updates KeystonePublic to KeystoneV3Public.
Additional information: From the perspective of the Octavia API log file, this is what the error looked like: 2019-03-27 19:34:03.042 1 DEBUG octavia.certificates.manager.barbican [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Setting project ACL for certificate secret... set_acls /usr/lib/python2.7/site-packages/octavia/certificates/manager/barbican.py:148 2019-03-27 19:34:03.747 1 DEBUG barbicanclient.client [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Creating Client object Client /usr/lib/python2.7/site-packages/barbicanclient/client.py:156 2019-03-27 19:34:03.809 1 DEBUG barbicanclient.v1.acls [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Getting ACL for secret href: http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/acl get /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:458 2019-03-27 19:34:04.106 1 DEBUG wsme.api [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222 In an attempt to get a more meaningful error message, I modified /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py and was able to get it to generate this traceback: 2019-03-28 04:44:31.080 1 ERROR root [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] : NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398) 2019-03-28 04:44:31.080 1 ERROR root Traceback (most recent call last): 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py", line 333, in load_acls_data 2019-03-28 04:44:31.080 1 ERROR root response = self._api.get(self.acl_ref) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 70, in get 2019-03-28 04:44:31.080 1 ERROR root return super(_HTTPClient, self).get(*args, **kwargs).json() 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get 2019-03-28 04:44:31.080 1 ERROR root return self.request(url, 'GET', **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 62, in request 2019-03-28 04:44:31.080 1 ERROR root resp = super(_HTTPClient, self).request(*args, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request 2019-03-28 04:44:31.080 1 ERROR root return self.session.request(url, method, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request 2019-03-28 04:44:31.080 1 ERROR root auth_headers = self.get_auth_headers(auth) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2019-03-28 04:44:31.080 1 ERROR root return auth.get_headers(self, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2019-03-28 04:44:31.080 1 ERROR root token = self.get_token(session) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2019-03-28 04:44:31.080 1 ERROR root return self.get_access(session).auth_token 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2019-03-28 04:44:31.080 1 ERROR root self.auth_ref = self.get_auth_ref(session) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref 2019-03-28 04:44:31.080 1 ERROR root return self._plugin.get_auth_ref(session, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref 2019-03-28 04:44:31.080 1 ERROR root authenticated=False, log=False) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post 2019-03-28 04:44:31.080 1 ERROR root return self.request(url, 'POST', **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request 2019-03-28 04:44:31.080 1 ERROR root raise exceptions.from_response(resp, method, url) 2019-03-28 04:44:31.080 1 ERROR root NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398) 2019-03-28 04:44:31.080 1 ERROR root 2019-03-28 04:44:31.152 1 DEBUG barbicanclient.v1.acls [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] After _api.get load_acls_data /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:336 2019-03-28 04:44:31.162 1 DEBUG wsme.api [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222
Kellen, thank you so much for the detailed report, root cause analysis and proposed patch! Since the auth URL is used between services, I think KeystoneV3Internal would be even more adequate. Let me know if you disagree. In the mean time I'll go ahead and propose the fix upstream.
KeystoneV3Internal makes sense to me. Thanks!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1738