Bug 1901916

Summary: Clean up legacy Cinder... "No route to host" in ipv6 deployments
Product: Red Hat OpenStack Reporter: Filip Hubík <fhubik>
Component: puppet-tripleoAssignee: Michele Baldessari <michele>
Status: CLOSED ERRATA QA Contact: David Rosenfeld <drosenfe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 16.2 (Train)CC: abishop, apevec, hjensas, jbuchta, jjoyce, jschluet, lmiccini, ltoscano, mburns, mgarciac, michele, ramishra, slinaber, tvignaud
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: puppet-tripleo-11.5.0-2.20201223234910.2b246e3.el8ost.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-15 07:10:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
error_isolated.log none

Description Filip Hubík 2020-11-26 12:22:21 UTC
Created attachment 1733717 [details]
error_isolated.log

Description of problem:
Since build RHOS-16.2-RHEL-8-20201125.n.1 we see failure in ipv6 enabled deployments, during OC deployment (after +-1hr runtime), on task "[Clean up legacy Cinder keystone catalog entries]":

...
Traceback (most recent call last):
  File \"<stdin>\", line 102, in <module>
  File \"<stdin>\", line 94, in _ansiballz_main
  File \"<stdin>\", line 40, in invoke_module
  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_os_keystone_service_payload_j0pf9n24/ansible_os_keystone_service_payload.zip/ansible/modules/cloud/openstack/os_keystone_service.py\", line 194, in <module>
  File \"/tmp/ansible_os_keystone_service_payload_j0pf9n24/ansible_os_keystone_service_payload.zip/ansible/modules/cloud/openstack/os_keystone_service.py\", line 153, in main
  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 510, in search_services
    services = self.list_services()
  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 485, in list_services
    if self._is_client_version('identity', 2):
  File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 459, in _is_client_version
    client = getattr(self, client_name)
  File \"/usr/lib/python3.6/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client
    'identity', min_version=2, max_version='3.latest')
  File \"/usr/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py\", line 406, in _get_versioned_client
    if adapter.get_endpoint():
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py\", line 282, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/session.py\", line 1200, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 380, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 271, in get_endpoint_data
    service_catalog = self.get_access(session).service_catalog
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py\", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 206, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File \"/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py\", line 161, in _do_create_plugin
    'auth_url is correct. %s' % e)
keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://[2620:52:0:13b8:5054:ff:fe3e:3]:5000: HTTPConnectionPool(host='2620:52:0:13b8:5054:ff:fe3e:3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb442ab978>: Failed to establish a new connection: [Errno 113] No route to host',))
", "module_stdout": "", "msg": "MODULE FAILURE
See stdout/stderr for the exact error", "rc": 1


Additional info:
(prettyfied full task-level isolated error log attached)
possibly from https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cinder/cinder-api-container-puppet.yaml#L418 ?

Comment 4 Luigi Toscano 2020-11-26 14:13:24 UTC
The new code comes from https://review.opendev.org/c/openstack/tripleo-heat-templates/+/757414 - but it doesn't seem to be doing anything nasty.

Comment 14 errata-xmlrpc 2021-09-15 07:10:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483