Description of problem: Hi, when updating from 16.0 to 16.1 a new parameter is required depending on "how you set up you tls configuration". For instance in infrared default tls setup we have: - virt/enable-tls.yaml generated using 16.0 templates; - re-used when doing the 16.1 update from 16.0 But if that file hasn't: - InternalTLSCAFile: '' set then the update fails during converge with: TASK [tripleo-keystone-resources : Create default domain] ********************** Monday 08 June 2020 11:48:15 +0000 (0:00:00.159) 0:33:35.907 *********** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ipa/ca.crt fatal: [undercloud]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_os_keystone_domain_payload_dbs6_ld4/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 185, in <module>\n File \"/tmp/ansible_os_keystone_domain_payload_dbs6_ld4/ansible_os_keystone_domain_payload.zip/ansible/modules/cloud/openstack/os_keystone_domain.py\", line 145, in main\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 883, in search_domains\n return self.list_domains(**filters)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 856, in list_domains\n data = self._identity_client.get(\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 406, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 282, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1200, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1442, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 526, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 101, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1098, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 888, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 979, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 646, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 416, in send\n self.cert_verify(conn, request.url, verify, cert)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 228, in cert_verify\n \"invalid path: {}\".format(cert_loc))\nOSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ipa/ca.crt\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} This is with the patch mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1840640 because even if the patch fixes for 16.1 deployment, it doesn't solve it for 16.0 updated env. So we need to make sure the user is aware that it need to update his/her configuration before doing an update.
Changing ownership to DFG:Security so that they can explain correctly when and how the user should change add the new parameter and setting it to empty.
Probably best to make ownership both Security and Upgrade. I'll most likely be working on it. So essentially, we just need to document the need for InternalTLSCAFile when doing a 16.0 to 16.1 update. Sounds fine to me. However, would this also affect upgrades too e.g. 13->16.1?
Also if I understand correctly, the main reason for the inclusion of the new param is because we used it for the undercloud only in 16.0, and now we're using this param for both the undercloud and overcloud?
Hey dan, so the context of when exactly you need to include that should be provided by security (got a call with them and it may depends on the tls setup). Now why do we need it is because the way security parameter is done/passed down in 16.1. So we need to adjust from 16.0 and certainly from 13.0 even if we should check that in CI as well. But again, I can test it part of it (basically what infrared gives me), but I don't have the detailed answer.
Dan, During the install process, some default users, projects and groups are created on the overcloud. These used to be executed on the controllers, but this had the unfortunate effect of requiring credentials to be added to the controllers. Instead, in 16.1, this operation was moved to the undercloud. The parameter InternalTLSCAFile provides the location of the CA cert bundle for the overcloud instance. The above operation will fail if this parameter is not set correctly. This is not an issue for new deployments because the the THT templates do set this parameter as needed. But if a customer tries to do an upgrade using old templates, then the defaults may not be correct. On new deployments, the parameter is typically set in one of two templates: environments/ssl/enable-tls.yaml environments/ssl/enable-internal-tls.j2.yaml The first template is typically used when a cert is provided to the deployment for the public interfaces. The second template is used when deploying tls-everywhere -- tls interfaces on both public and private interfaces. For upgrades, therefore, the guidance is as follows: -- If your deployment is using public TLS - ie. TLS on the public interfaces only, then you must have added the required cert to the deployment somewhere and trusted it in the default trust store. The easiest solution here is to set this parameter to '' (empty string). This has the effect of telling the undercloud to use the certificates in the default trust store. -- If your deployment is using TLS-Everywhere and your public certs are being issued by IPA, then the parameter should be set to /etc/ipa/ca.crt. Actually -- in this case -- given that the IPA CA cert should already be in the default trust store -- it should be OK to use '' in this case too. Leaving for Rabi to comment on this. This parameter should be set prior to upgrade or the above keystone operations on the underclouds will fail. This is going to be required for any upgrade/update to 16.1. Rabi, does this match your understanding of things?
*** Bug 1849703 has been marked as a duplicate of this bug. ***
Stealing this BZ so I can include this information in the update and upgrade guides.
Hi Dan, what is the current state of that bugzilla. It looks "solved" to me, so I'm wondering what is needed to close it ? Thanks,
Correct. This has been implemented: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/keeping_red_hat_openstack_platform_updated/preparing-for-a-minor-update#updating-your-tls-ssl-configuration BZ is safe to close.