The issue here is likely an incorrect default setting when used within the context of
public TLS.  /etc/ipa/ca.crt should only be used when TLS-E is enabled.

The issue is likely due to a change that was made to write the cacert to clouds.yaml
as from this commit:

https://review.opendev.org/#/c/727959/

quickstart overrides the setting for InternalTLSCAFile

https://review.opendev.org/#/c/728358/2/roles/overcloud-ssl/library/tls_tht.py

Its likely that infrared does not override this setting.

We need to make sure this is documented correctly as well.

*** Bug 1843404 has been marked as a duplicate of this bug. ***

hi,

Update from 16.0 latest_cdn to 16.1 RHOS-16.1-RHEL-8-20200602.n.1 puddle: show the original error during converge.  

Moving this back to ON_DEV as :

(undercloud) [stack@undercloud-0 ~]$ rpm -qa | grep openstack-tripleo-heat-templates-11.3.2-0.20200530033438.0dfce4e.el8ost
openstack-tripleo-heat-templates-11.3.2-0.20200530033438.0dfce4e.el8ost.noarch

and I can see that https://review.opendev.org/#/c/731386/ is included in there 

and we still have the original issue.

Testing if https://review.opendev.org/#/c/733036/2 helps here.

Not that in the job we have ir_tripleo_undercloud_ssl set to true.

Hi,

This is not working on update from 16.0 to 16.1 using standard infrared deployment and update.

I've tested that patch as it's in the latest puddle already:

(undercloud) [stack@undercloud-0 ~]$ rpm -qa | grep heat-templates
openstack-tripleo-heat-templates-11.3.2-0.20200603183438.7b2c249.el8ost.noarch

(undercloud) [stack@undercloud-0 ~]$ sudo grep InternalTLS /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml
InternalTLSCAFile: ''


and we still have:


TASK [tripleo-keystone-resources : Create default domain] **********************
Monday 08 June 2020  11:48:15 +0000 (0:00:00.159)       0:33:35.907 ***********
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ipa/ca.crt
fatal: [undercloud]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_os_keystone_domain_payload_dbs6_ld4/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 185, in <module>\n  File \"/tmp/ansible_os_keystone_domain_payload_dbs6_ld4/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 145, in main\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 883, in search_domains\n    return self.list_domains(**filters)\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 856, in list_domains\n    data = self._identity_client.get(\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n    'identity', min_version=2, max_version='3.latest')\n  File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 406, in _get_versioned_client\n    if adapter.get_endpoint():\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 282, in get_endpoint\n    return self.session.get_endpoint(auth or self.auth, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1200, in get_endpoint\n    return auth.get_endpoint(self, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n    allow_version_hack=allow_version_hack, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n    service_catalog = self.get_access(session).service_catalog\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n    self.auth_ref = self.get_auth_ref(session)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n    self._plugin = self._do_create_plugin(session)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n    authenticated=False)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n    authenticated=authenticated)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1442, in get_discovery\n    disc = Discover(session, url, authenticated=authenticated)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 526, in __init__\n    authenticated=authenticated)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 101, in get_version_data\n    resp = session.get(url, headers=headers, authenticated=authenticated)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1098, in get\n    return self.request(url, 'GET', **kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 888, in request\n    resp = send(**kwargs)\n  File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 979, in _send_request\n    resp = self.session.request(method, url, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 416, in send\n    self.cert_verify(conn, request.url, verify, cert)\n  File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 228, in cert_verify\n    \"invalid path: {}\".format(cert_loc))\nOSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ipa/ca.crt\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

during overcloud update converge.


Note that is an update from 16.0 16.1 done using infrared.

We would need help to debug that issue, we have a live env.

Hi,

so just to be very clear here, it seems that without manual intervention where the user coming from 16.0.{GA,1,2} explicitly add this new parameter:

  InternalTLSCAFile: ''

to his/her heat template tls definition, then the update from 16.0 will fails during converge.

We need:

 1. document the required change: https://bugzilla.redhat.com/show_bug.cgi?id=1845091
 2. fix the ci tooling (tripleo-upgrade);

This bug, from the point of view of update is still an issue until the previous two points are done.

We should also have a very explicit errata note from this bug for people updating from 16.0.

This issue has been addressed in code.

There has been a documentation bz created for the new parameter:

https://bugzilla.redhat.com/show_bug.cgi?id=1845091

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148