Description of problem: 2022-05-23 23:27:25.890169 | 48d539a1-1679-937c-7762-0000000000fc | OK | External deployment step 4 | undercloud -> localhost | result={ "changed": false, "msg": "Use --start-at-task 'External deployment step 4' to resume from this task" } [WARNING]: ('undercloud -> localhost', '48d539a1-1679-937c-7762-0000000000fc') missing from stats 2022-05-23 23:27:25.967712 | 48d539a1-1679-937c-7762-0000000000fd | TIMING | include_tasks | undercloud | 0:49:41.832294 | 0.05s 2022-05-23 23:27:25.992708 | 4175f2a2-4f95-4c9f-88d0-e5f25845a384 | INCLUDED | /home/stack/overcloud-deploy/overcloud/config-download/overcloud/external_deploy_steps_tasks_step4.yaml | undercloud 2022-05-23 23:27:26.016756 | 48d539a1-1679-937c-7762-000000012cb0 | TASK | Clean up legacy Cinder keystone catalog entries out=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 727, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1026, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in <module>\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 141, in run\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_r2022-05-23 23:27:29.289019 | 48d539a1-1679-937c-7762-000000012cb0 | FATAL | Clean up legacy Cinder keystone catalog entries | undercloud | item={'service_name': 'cinderv2', 'service_type': 'volumev2'} | error={"ansible_index_var": "cinder_api_service", "ansible_loop_var": "item", "changed": false, "cinder_api_service": 0, "item": {"service_name": "cinderv2", "service_type": "volumev2"}, "module_stderr": "Failed to discover available identity versions when contacting https://overcloud.domain.com:13000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 677, in urlopen\n chunked=chunked,\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 381, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 978, in _validate_conn\n conn.connect()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 371, in connect\n ssl_context=context,\n File \"/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 384, in ssl_wrap_socket\n return context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\", line 365, in wrap_socket\n _context=self, _session=session)\n File \"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n self.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in do_handshake\n self._sslobj.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 648, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 727, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1026, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in <module>\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_t4zsbz1z/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 141, in run\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 161, in _do_create_plugin\n 'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2022-05-23 23:27:29.292242 | 48d539a1-1679-937c-7762-000000012cb0 | TIMING | Clean up legacy Cinder keystone catalog entries | undercloud | 0:49:45.156843 | 3.27s timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 727, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1026, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in <module>\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 141, in run\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref =2022-05-23 23:27:32.139526 | 48d539a1-1679-937c-7762-000000012cb0 | FATAL | Clean up legacy Cinder keystone catalog entries | undercloud | item={'service_name': 'cinderv3', 'service_type': 'volume'} | error={"ansible_index_var": "cinder_api_service", "ansible_loop_var": "item", "changed": false, "cinder_api_service": 1, "item": {"service_name": "cinderv3", "service_type": "volume"}, "module_stderr": "Failed to discover available identity versions when contacting https://overcloud.domain.com:13000. Attempting to parse version from URL.\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 677, in urlopen\n chunked=chunked,\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 381, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 978, in _validate_conn\n conn.connect()\n File \"/usr/lib/python3.6/site-packages/urllib3/connection.py\", line 371, in connect\n ssl_context=context,\n File \"/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py\", line 384, in ssl_wrap_socket\n return context.wrap_socket(sock, server_hostname=server_hostname)\n File \"/usr/lib64/python3.6/ssl.py\", line 365, in wrap_socket\n _context=self, _session=session)\n File \"/usr/lib64/python3.6/ssl.py\", line 776, in __init__\n self.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 1036, in do_handshake\n self._sslobj.do_handshake()\n File \"/usr/lib64/python3.6/ssl.py\", line 648, in do_handshake\n self._sslobj.do_handshake()\nssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 449, in send\n timeout=timeout\n File \"/usr/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 727, in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]\n File \"/usr/lib/python3.6/site-packages/urllib3/util/retry.py\", line 439, in increment\n raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1022, in _send_request\n resp = self.session.request(method, url, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 542, in request\n resp = self.send(prep, **send_kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/sessions.py\", line 655, in send\n r = adapter.send(request, **kwargs)\n File \"/usr/lib/python3.6/site-packages/requests/adapters.py\", line 514, in send\n raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 138, in _do_create_plugin\n authenticated=False)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 610, in get_discovery\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 1452, in get_discovery\n disc = Discover(session, url, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 536, in __init__\n authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/discover.py\", line 102, in get_version_data\n resp = session.get(url, headers=headers, authenticated=authenticated)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1141, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 931, in request\n resp = send(**kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1026, in _send_request\n raise exceptions.SSLError(msg)\nkeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 185, in <module>\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 181, in main\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/module_utils/openstack.py\", line 407, in __call__\n File \"/tmp/ansible_openstack.cloud.catalog_service_payload_hcif51qb/ansible_openstack.cloud.catalog_service_payload.zip/ansible_collections/openstack/cloud/plugins/modules/catalog_service.py\", line 141, in run\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 517, in search_services\n services = self.list_services()\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 492, in list_services\n if self._is_client_version('identity', 2):\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 460, in _is_client_version\n client = getattr(self, client_name)\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 407, in _get_versioned_client\n if adapter.get_endpoint():\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 291, in get_endpoint\n return self.session.get_endpoint(auth or self.auth, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1243, in get_endpoint\n return auth.get_endpoint(self, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint\n allow_version_hack=allow_version_hack, **kwargs)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data\n service_catalog = self.get_access(session).service_catalog\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access\n self.auth_ref = self.get_auth_ref(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref\n self._plugin = self._do_create_plugin(session)\n File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 161, in _do_create_plugin\n 'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} Version-Release number of selected component (if applicable): Openstack wallaby How reproducible: Install wallaby with SSL Steps to Reproduce: 1. Complete basic requirements for deployment 2. Enable ssl using enable-tls.yaml, inject-ca-heira.yaml and cloudname.yaml Actual results: Overcloud deployed with ssl Expected results: keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://overcloud.domain.com:13000: HTTPSConnectionPool(host='overcloud.domain.com', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} Additional info:
Hi, Are you using inject-trust-anchor.yaml: https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/environments/ssl/inject-trust-anchor.yaml You mention inject-ca-hiera.yaml, but we don't ship a file with that name in Wallaby. The problem you're experiencing seems to indicate that the CA isn't being inserted into the trusts of the overcloud nodes. So I suspect that file you have used is incorrect. You can find documentation relevant to tripleo Wallaby: https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/features/ssl.html#overcloud-ssl Specifically, this section: https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/features/ssl.html#certificate-details I also covered all of the steps in that documentation in some detail in this video that you might find helpful: https://www.youtube.com/watch?v=FmO6n1fUiYU
Hi, I can't edit the comments either. But I have marked them as private for you. Regarding your last comment, no that shouldn't be an issue. It is trying to hit: https://overcloud.domain.com:13000 and failing to verify the SSL certificate. There will be a /etc/hosts entry for overcloud.domain.com so DNS shouldn't be relevant in this case. The problem is still that the overcloud nodes don't trust the CA certificate. Are you able to confirm which version of RHOSP you are running. This BZ has been opened against RHOSP17. Can you share the output from: cat /etc/rhosp-release And also, are you able to tar up all of the templates you are using and attach them to this BZ? I see you have used: THT=/usr/share/openstack-tripleo-heat-templates/ > -e $THT/environments/ssl/enable-tls.yaml \ > -e $THT/environments/ssl/tls-endpoints-public-dns.yaml \ > -e $THT/environments/ssl/inject-trust-anchor.yaml \ Are you adding the SSL certificates to these files before including them in the overcloud deploy command? Like this for example is my inject-trust-anchor.yaml file and you can see my CA certificate has been included there: https://github.com/r3d3mpt10n/tripleo-home/blob/main/inject-trust-anchor.yaml#L16-L41 If you can please attach all of the templates you're using we can take a look. Please also confirm which version of RHOSP you are running by executing the following on the undercloud node: cat /etc/rhosp-release
Ah ok, I see. No worries. Upstream bugs for tripleo should be raised here instead: https://bugs.launchpad.net/tripleo It looks like that CA cert isn't able to verify the TLS certificate that you're trying to use: ❯ openssl verify -verbose -CAfile ca.pem cert.pem C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA error 2 at 1 depth lookup: unable to get issuer certificate error cert.pem: verification failed Where ca.pem is taken from your SSLRootCertificate option and cert.pem was taken from you SSLCertificate option. So I believe this is the issue. Is SSLRootCertificate the same as the contents of this file? /etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt I assume that is the CA you're trying to use in this scenario. If I run that same command using the certificates deployed on my overcloud controller node, we can see that it returns OK for the verification: [root@overcloud-controller-0 ~]# openssl verify -verbose -CAfile /etc/pki/ca-trust/source/anchors/ca.crt.pem /etc/pki/tls/private/overcloud_endpoint.pem /etc/pki/tls/private/overcloud_endpoint.pem: OK You should be able to run the same command on your overcloud controller, and I believe it will fail. This would indicate that either the CA certificate you have provided is incorrect, or the tls certificate used for overcloud_endpoint.pem was not signed by that CA Cert.
Hi, > Is SSLRootCertificate the same as the contents of this file? /etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt Yes That is the CA certificate i got from the vendor. There is a file /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem it contains the content of /etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt as one of many entries, when i use that to verify the cert, it succeeds. But when i try verifying using that content (SectigoRSADomainValidationSecureServerCA.crt) only it shows verification failed. [root@overcloud-controller-0 ~]# openssl verify -verbose -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/tls/private/overcloud_endpoint.pem /etc/pki/tls/private/overcloud_endpoint.pem: OK [root@overcloud-controller-0 ~]# Certificate Information: Common Name: *.bdxworld.com Subject Alternative Names: *.bdxworld.com, bdxworld.com Valid From: April 17, 2022 Valid To: April 18, 2023 Issuer: Sectigo RSA Domain Validation Secure Server CA, Sectigo Limited Write review of Sectigo Serial Number: 213d77b4307f267b577c0243c6f971d0
I suspect that this is the problem then: > But when i try verifying using that content (SectigoRSADomainValidationSecureServerCA.crt) only it shows verification failed. Because you are using this option: # Specifies the default CA cert to use if TLS is used for services in the public network. # Type: string PublicTLSCAFile: /etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt That will add `/etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt` to your clouds.yaml file here: [stack@tripleo-director ~]$ grep cacert /home/stack/.config/openstack/clouds.yaml cacert: /etc/pki/ca-trust/source/anchors/overcloud-cacert.pem cacert: /etc/pki/ca-trust/source/anchors/overcloud-cacert.pem So you could verify if that is indeed the issue by doing this: $ su - stack $ OS_CLOUD=overcloud openstack catalog list Do you get the same error? What happens now if you change that cacert argument in the clouds.yaml file to the following under the overcloud: section: ``` cacert: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ``` The try again using: ``` OS_CLOUD=overcloud openstack catalog list ``` Do you still get an error, or does that work now?
(overcloud) [stack@hkg2director ~]$ OS_CLOUD=overcloud openstack catalog list /usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change. To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`. DeprecationWarning Could not find a suitable TLS CA certificate bundle, invalid path: /etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt (overcloud) [stack@hkg2director ~]$ vi .config/openstack/clouds.yaml (overcloud) [stack@hkg2director ~]$ vi .config/openstack/clouds.yaml (overcloud) [stack@hkg2director ~]$ OS_CLOUD=overcloud openstack catalog list /usr/lib64/python3.6/site-packages/_yaml/__init__.py:23: DeprecationWarning: The _yaml extension module is now located at yaml._yaml and its location is subject to change. To use the LibYAML-based parser and emitter, import from `yaml`: `from yaml import CLoader as Loader, CDumper as Dumper`. DeprecationWarning +-----------+----------------+--------------------------------------------------------------------------------------+ | Name | Type | Endpoints | +-----------+----------------+--------------------------------------------------------------------------------------+ | aodh | alarming | regionOne | | | | internal: http://172.25.201.250:8042 | | | | regionOne | | | | public: http://172.25.201.150:8042 | | | | regionOne | | | | admin: http://172.25.201.250:8042 | | | | | | placement | placement | regionOne | | | | admin: http://172.25.201.250:8778/placement | | | | regionOne | | | | internal: http://172.25.201.250:8778/placement | | | | regionOne | | | | public: http://172.25.201.150:8778/placement | | | | | | gnocchi | metric | regionOne | | | | public: http://172.25.201.150:8041 | | | | regionOne | | | | internal: http://172.25.201.250:8041 | | | | regionOne | | | | admin: http://172.25.201.250:8041 | | | | | | glance | image | regionOne | | | | internal: http://172.25.201.250:9292 | | | | regionOne | | | | admin: http://172.25.201.250:9292 | | | | regionOne | | | | public: http://172.25.201.150:9292 | | | | | | keystone | identity | regionOne | | | | internal: http://172.25.201.250:5000 | | | | regionOne | | | | admin: http://172.25.201.250:35357 | | | | regionOne | | | | public: https://overcloud.bdxworld.com:13000 | | | | | | heat-cfn | cloudformation | regionOne | | | | internal: http://172.25.201.250:8000/v1 | | | | regionOne | | | | public: http://172.25.201.150:8000/v1 | | | | regionOne | | | | admin: http://172.25.201.250:8000/v1 | | | | | | neutron | network | regionOne | | | | admin: http://172.25.201.250:9696 | | | | regionOne | | | | public: http://172.25.201.150:9696 | | | | regionOne | | | | internal: http://172.25.201.250:9696 | | | | | | heat | orchestration | regionOne | | | | internal: http://172.25.201.250:8004/v1/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | public: http://172.25.201.150:8004/v1/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | admin: http://172.25.201.250:8004/v1/5d922243077045c48fe4b075e386551b | | | | | | octavia | load-balancer | regionOne | | | | public: http://172.25.201.150:9876 | | | | regionOne | | | | admin: http://172.25.201.250:9876 | | | | regionOne | | | | internal: http://172.25.201.250:9876 | | | | | | cinderv3 | volumev3 | regionOne | | | | internal: http://172.25.201.250:8776/v3/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | public: http://172.25.201.150:8776/v3/5d922243077045c48fe4b075e386551b | | | | regionOne | | | | admin: http://172.25.201.250:8776/v3/5d922243077045c48fe4b075e386551b | | | | | | swift | object-store | regionOne | | | | public: http://172.25.201.150:8080/swift/v1/AUTH_5d922243077045c48fe4b075e386551b | | | | regionOne | | | | admin: http://172.25.202.50:8080/swift/v1/AUTH_5d922243077045c48fe4b075e386551b | | | | regionOne | | | | internal: http://172.25.202.50:8080/swift/v1/AUTH_5d922243077045c48fe4b075e386551b | | | | | | nova | compute | regionOne | | | | admin: http://172.25.201.250:8774/v2.1 | | | | regionOne | | | | internal: http://172.25.201.250:8774/v2.1 | | | | regionOne | | | | public: http://172.25.201.150:8774/v2.1 | | | | | +-----------+----------------+--------------------------------------------------------------------------------------+ (overcloud) [stack@hkg2director ~]$ > So you could verify if that is indeed the issue by doing this: > > $ su - stack > $ OS_CLOUD=overcloud openstack catalog list > > Do you get the same error? Answer: getting error > What happens now if you change that cacert argument in the clouds.yaml file to the following under the overcloud: section: > > ``` > cacert: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem > ``` > > The try again using: > ``` > OS_CLOUD=overcloud openstack catalog list > ``` > >Do you still get an error, or does that work now? Answer: Working fine, No error So should i involve sectigo(my ssl provider) to for a new and correct CA cert? or should i use the contents of the whole bundle /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem in inject-trust-anchor.yaml (which i think is not feasible and is wrong)
Ah perfect. Na, the only change you need to make is to your /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml template: Change this: PublicTLSCAFile: /etc/pki/ca-trust/source/anchors/SectigoRSADomainValidationSecureServerCA.crt To this: PublicTLSCAFile: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem That will solve your problem. As a side note though, it's not advisable to directly edit templates in /usr/share/openstack-tripleo-heat-templates as these changes will all be reverted during updates. You should copy the files that you want to change into a different directory and then include them in your overcloud deploy command to avoid having your changes reverted.
Hi, Thank you for your help. The issue is now resolved and my overcloud is now running with SSL enabled. Thank you so much. Should I close this stating not a bug?
Excellent, glad to hear it's now working for you. Yeah, I will close this as not a bug now. Nice to work with you, all the best with you deployment.