Bug 1693808

Summary: Creating TERMINATED_HTTPS Octavia loadbalancer listener fails with 'Could not retrieve certificate' after z5 update
Product: Red Hat OpenStack Reporter: Kellen Gattis <kgattis>
Component: openstack-tripleo-heat-templatesAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: akaris, asimonel, astafeye, cgoncalves, ealcaniz, jbiao, mburns, philippe.cyr, pmannidi, sputhenp
Target Milestone: z7Keywords: Regression, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.3.1-37.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1716801 (view as bug list) Environment:
Last Closed: 2019-07-10 13:03:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1716801    
Bug Blocks: 1569129    
Attachments:
Description Flags
Updates octavia-base to use v3 endpoint for service_auth url none

Description Kellen Gattis 2019-03-28 16:34:50 UTC
Created attachment 1549132 [details]
Updates octavia-base to use v3 endpoint for service_auth url

Description of problem:
After updating to the latest z5 release of RHOSP 13, trying to create a TERMINATED_HTTPS loadbalancer listener is failing with 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request.


Version-Release number of selected component (if applicable):
registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-66
python-octavia-2.0.3-2.el7ost.noarch
puppet-octavia-12.4.0-8.el7ost.noarch
openstack-octavia-common-2.0.3-2.el7ost.noarch
openstack-octavia-api-2.0.3-2.el7ost.noarch
openstack-tripleo-heat-templates-8.2.0-6.1.el7ost.noarch

How reproducible:
Easy to reproduce.
Following the same series of steps against the scenario when registry.access.redhat.com/rhosp13/openstack-octavia-api:13.0-64 is the image used by octavia_api works, but fails with 13.0-66.

Steps to Reproduce:
Standard steps for creating a TERMINATED_HTTPS loadbalancer -

openssl pkcs12 -export -inkey server.key -in server.crt -certfile ca-chain.crt -passout pass: -out server.p12
openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)"
openstack acl user add -u <octavia_user_uuid> $(openstack secret list | awk '/ tls_secret1 / {print $2}')
openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}') lb1

Actual results:
Creation fails with: 'Could not retrieve certificate: ['<HATEOAS>'] (HTTP 400) (Request-ID: req-<uuid>)' where <HATEOAS> is the secret URL and <uuid> is the uuid of the request

Expected results:
Loadbalancer listener is created

Additional info:
The new version of the Octavia image looks like it includes some enhancements that are designed to automatically add the octavia user uuid to the barbican secret containing the certificate data
Presumably, this is related to this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1569129

From what I can tell, the issue with creating a TERMINATED_HTTPS listener against the new code is that the auth_url in the [service_auth] section of octavia.conf is being configured by tripleo to use KeystonePublic as the endpoint and not KeystoneV3Public.
Hand-editing octavia.conf and adding "/v3" to the auth_url, restarting the service, and running through the 'steps to reproduce' results in a successful loadbalancer listener creation.



The value for auth_url comes from the following line in /usr/share/openstack-tripleo-heat-templates/puppet/services/octavia-base.yaml:
             octavia::service_auth::auth_url: {get_param: [EndpointMap, KeystonePublic, uri]}


I've attached a patch that simply updates KeystonePublic to KeystoneV3Public.

Comment 1 Kellen Gattis 2019-03-28 17:31:21 UTC
Additional information:

From the perspective of the Octavia API log file, this is what the error looked like:

2019-03-27 19:34:03.042 1 DEBUG octavia.certificates.manager.barbican [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Setting project ACL for certificate secret... set_acls /usr/lib/python2.7/site-packages/octavia/certificates/manager/barbican.py:148
2019-03-27 19:34:03.747 1 DEBUG barbicanclient.client [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Creating Client object Client /usr/lib/python2.7/site-packages/barbicanclient/client.py:156
2019-03-27 19:34:03.809 1 DEBUG barbicanclient.v1.acls [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Getting ACL for secret href: http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/acl get /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:458
2019-03-27 19:34:04.106 1 DEBUG wsme.api [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222



In an attempt to get a more meaningful error message, I modified /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py and was able to get it to generate this traceback:
2019-03-28 04:44:31.080 1 ERROR root [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] : NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398)
2019-03-28 04:44:31.080 1 ERROR root Traceback (most recent call last):
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py", line 333, in load_acls_data
2019-03-28 04:44:31.080 1 ERROR root     response = self._api.get(self.acl_ref)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 70, in get
2019-03-28 04:44:31.080 1 ERROR root     return super(_HTTPClient, self).get(*args, **kwargs).json()
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2019-03-28 04:44:31.080 1 ERROR root     return self.request(url, 'GET', **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 62, in request
2019-03-28 04:44:31.080 1 ERROR root     resp = super(_HTTPClient, self).request(*args, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request
2019-03-28 04:44:31.080 1 ERROR root     return self.session.request(url, method, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request
2019-03-28 04:44:31.080 1 ERROR root     auth_headers = self.get_auth_headers(auth)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers
2019-03-28 04:44:31.080 1 ERROR root     return auth.get_headers(self, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2019-03-28 04:44:31.080 1 ERROR root     token = self.get_token(session)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2019-03-28 04:44:31.080 1 ERROR root     return self.get_access(session).auth_token
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-03-28 04:44:31.080 1 ERROR root     self.auth_ref = self.get_auth_ref(session)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref
2019-03-28 04:44:31.080 1 ERROR root     return self._plugin.get_auth_ref(session, **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref
2019-03-28 04:44:31.080 1 ERROR root     authenticated=False, log=False)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post
2019-03-28 04:44:31.080 1 ERROR root     return self.request(url, 'POST', **kwargs)
2019-03-28 04:44:31.080 1 ERROR root   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request
2019-03-28 04:44:31.080 1 ERROR root     raise exceptions.from_response(resp, method, url)
2019-03-28 04:44:31.080 1 ERROR root NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398)
2019-03-28 04:44:31.080 1 ERROR root
2019-03-28 04:44:31.152 1 DEBUG barbicanclient.v1.acls [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] After _api.get load_acls_data /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:336
2019-03-28 04:44:31.162 1 DEBUG wsme.api [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222

Comment 2 Carlos Goncalves 2019-04-15 08:49:31 UTC
Kellen, thank you so much for the detailed report, root cause analysis and proposed patch!

Since the auth URL is used between services, I think KeystoneV3Internal would be even more adequate. Let me know if you disagree. In the mean time I'll go ahead and propose the fix upstream.

Comment 6 Kellen Gattis 2019-04-16 13:33:48 UTC
KeystoneV3Internal makes sense to me.

Thanks!

Comment 26 errata-xmlrpc 2019-07-10 13:03:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1738