Bug 1693808
Summary: | Creating TERMINATED_HTTPS Octavia loadbalancer listener fails with 'Could not retrieve certificate' after z5 update | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Kellen Gattis <kgattis> | ||||
Component: | openstack-tripleo-heat-templates | Assignee: | Carlos Goncalves <cgoncalves> | ||||
Status: | CLOSED ERRATA | QA Contact: | Bruna Bonguardo <bbonguar> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 13.0 (Queens) | CC: | akaris, asimonel, astafeye, cgoncalves, ealcaniz, jbiao, mburns, philippe.cyr, pmannidi, sputhenp | ||||
Target Milestone: | z7 | Keywords: | Regression, Triaged, ZStream | ||||
Target Release: | 13.0 (Queens) | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | openstack-tripleo-heat-templates-8.3.1-37.el7ost | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | |||||||
: | 1716801 (view as bug list) | Environment: | |||||
Last Closed: | 2019-07-10 13:03:20 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | 1716801 | ||||||
Bug Blocks: | 1569129 | ||||||
Attachments: |
|
Description
Kellen Gattis
2019-03-28 16:34:50 UTC
Additional information: From the perspective of the Octavia API log file, this is what the error looked like: 2019-03-27 19:34:03.042 1 DEBUG octavia.certificates.manager.barbican [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Setting project ACL for certificate secret... set_acls /usr/lib/python2.7/site-packages/octavia/certificates/manager/barbican.py:148 2019-03-27 19:34:03.747 1 DEBUG barbicanclient.client [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Creating Client object Client /usr/lib/python2.7/site-packages/barbicanclient/client.py:156 2019-03-27 19:34:03.809 1 DEBUG barbicanclient.v1.acls [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Getting ACL for secret href: http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/acl get /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:458 2019-03-27 19:34:04.106 1 DEBUG wsme.api [req-3d3432a4-04e3-4539-9dd2-ff00957e8aaf - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222 In an attempt to get a more meaningful error message, I modified /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py and was able to get it to generate this traceback: 2019-03-28 04:44:31.080 1 ERROR root [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] : NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398) 2019-03-28 04:44:31.080 1 ERROR root Traceback (most recent call last): 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py", line 333, in load_acls_data 2019-03-28 04:44:31.080 1 ERROR root response = self._api.get(self.acl_ref) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 70, in get 2019-03-28 04:44:31.080 1 ERROR root return super(_HTTPClient, self).get(*args, **kwargs).json() 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get 2019-03-28 04:44:31.080 1 ERROR root return self.request(url, 'GET', **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/barbicanclient/client.py", line 62, in request 2019-03-28 04:44:31.080 1 ERROR root resp = super(_HTTPClient, self).request(*args, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request 2019-03-28 04:44:31.080 1 ERROR root return self.session.request(url, method, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request 2019-03-28 04:44:31.080 1 ERROR root auth_headers = self.get_auth_headers(auth) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2019-03-28 04:44:31.080 1 ERROR root return auth.get_headers(self, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2019-03-28 04:44:31.080 1 ERROR root token = self.get_token(session) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2019-03-28 04:44:31.080 1 ERROR root return self.get_access(session).auth_token 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2019-03-28 04:44:31.080 1 ERROR root self.auth_ref = self.get_auth_ref(session) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref 2019-03-28 04:44:31.080 1 ERROR root return self._plugin.get_auth_ref(session, **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref 2019-03-28 04:44:31.080 1 ERROR root authenticated=False, log=False) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post 2019-03-28 04:44:31.080 1 ERROR root return self.request(url, 'POST', **kwargs) 2019-03-28 04:44:31.080 1 ERROR root File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request 2019-03-28 04:44:31.080 1 ERROR root raise exceptions.from_response(resp, method, url) 2019-03-28 04:44:31.080 1 ERROR root NotFound: (http://XX.XX.XX.XX:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-bcceb9c5-e71b-4a91-a33a-076dccf75398) 2019-03-28 04:44:31.080 1 ERROR root 2019-03-28 04:44:31.152 1 DEBUG barbicanclient.v1.acls [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] After _api.get load_acls_data /usr/lib/python2.7/site-packages/barbicanclient/v1/acls.py:336 2019-03-28 04:44:31.162 1 DEBUG wsme.api [req-286cec81-91a3-43cc-9532-f33acdd6302f - ddf3414ef08d48f2ae336cb64ed9313f - default default] Client-side error: Could not retrieve certificate: ['http://XX.XX.XX.XX:9311/v1/secrets/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'] format_exception /usr/lib/python2.7/site-packages/wsme/api.py:222 Kellen, thank you so much for the detailed report, root cause analysis and proposed patch! Since the auth URL is used between services, I think KeystoneV3Internal would be even more adequate. Let me know if you disagree. In the mean time I'll go ahead and propose the fix upstream. KeystoneV3Internal makes sense to me. Thanks! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1738 |