Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1783397

Summary: OSP13 undercloud upgrade fails with error - Failed to discover available identity versions when contacting https://X.X.X.X:13000/. Attempting to parse version from URL.
Product: Red Hat OpenStack Reporter: Chris Janiszewski <cjanisze>
Component: instack-undercloudAssignee: Raildo Mascena de Sousa Filho <rmascena>
Status: CLOSED WORKSFORME QA Contact: Arik Chernetsky <achernet>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: alee, jfrancoa, kbhaskar, mbollo, mburns, sbandyop, sgolovat
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-29 14:23:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
full undercloud log
none
undercloud.conf none

Description Chris Janiszewski 2019-12-13 16:53:24 UTC 
Created attachment 1644889 [details]
full undercloud log

Description of problem:
Attempted to upgrade undercloud - minor update (this is through multiple Z releases).

Invoked the following:
"openstack undercloud upgrade"

It looks like during the process the pem files have changed for the undercloud and the installer failed with the following:

2019-12-13 11:21:31,084 INFO: Failed to discover available identity versions when contacting https://172.31.8.10:13000/. Attempting to parse version from URL.
2019-12-13 11:21:31,084 INFO: Could not determine a suitable URL for the plugin
2019-12-13 11:21:31,133 INFO: + openstack quota set --cores -1 --instances -1 --ram -1
2019-12-13 11:21:31,878 INFO: Failed to discover available identity versions when contacting https://172.31.8.10:13000/. Attempting to parse version from URL.
2019-12-13 11:21:31,878 INFO: Could not determine a suitable URL for the plugin
2019-12-13 11:21:31,925 INFO: [2019-12-13 11:21:31,924] (os-refresh-config) [ERROR] during post-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/post-configure.d']' returned non-zero exit status 1]
2019-12-13 11:21:31,925 INFO: 
2019-12-13 11:21:31,925 INFO: [2019-12-13 11:21:31,924] (os-refresh-config) [ERROR] Aborting...
2019-12-13 11:21:31,933 DEBUG: An exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2441, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1618, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 675, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
2019-12-13 11:21:31,934 ERROR: 
#############################################################################
Undercloud upgrade failed.


Inside the logs I also see the following:
2019-06-05 10:08:52,984 INFO: ESC[mNotice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[undercloud-haproxy-public]/Concat[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/File[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/content: content changed '{md5}2a6fca6c97449449d63f767e4f655c05' to '{md5}f455717fdd89d32be6e7e24b9bd52334'ESC[0m
2019-06-05 10:08:52,985 INFO: ESC[mNotice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[undercloud-haproxy-public]/Concat[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/File[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/owner: owner changed 'root' to 'haproxy'ESC[0m
2019-06-05 10:08:52,985 INFO: ESC[mNotice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[undercloud-haproxy-public]/Concat[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/File[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/group: group changed 'root' to 'haproxy'ESC[0m
2019-06-05 10:08:52,986 INFO: ESC[mNotice: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Haproxy[undercloud-haproxy-public]/Concat[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/File[/etc/pki/tls/certs/undercloud-172.31.8.10.pem]/mode: mode changed '0600' to '0640'ESC[0m

Is this normal during the upgrade ?


Version-Release number of selected component (if applicable):
OSP13


How reproducible:


Steps to Reproduce:
1. https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/keeping_red_hat_openstack_platform_updated/assembly-upgrading_the_undercloud#performing-a-minor-update-of-an-undercloud


Actual results:


Expected results:


Additional info:
Comment 1 Chris Janiszewski 2019-12-13 16:54:06 UTC 
Created attachment 1644890 [details]
undercloud.conf
Comment 2 Chris Janiszewski 2019-12-13 16:55:43 UTC 
Also the undercloud no longer works:

(undercloud) [stack@undercloud-osp13 ~]$ openstack server list --debug
START with options: [u'server', u'list', u'--debug']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='https://172.31.8.10:13000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.34', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://172.31.8.10:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://172.31.8.10:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
messaging API version 2, cmd group openstack.messaging.v2
orchestration API version 1, cmd group openstack.orchestration.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
baremetal API version 1.34, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
dns API version 2, cmd group openstack.dns.v2
database API version 1, cmd group openstack.database.v1
alarming API version 2, cmd group openstack.alarming.v2
key_manager API version 1, cmd group openstack.key_manager.v1
container_infra API version 1, cmd group openstack.container_infra.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
metric API version 1, cmd group openstack.metric.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://172.31.8.10:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://172.31.8.10:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
command: server list -> openstackclient.compute.v2.server.ListServer (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'https://172.31.8.10:13000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://172.31.8.10:13000/', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET https://172.31.8.10:13000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"
Starting new HTTPS connection (1): 172.31.8.10
Failed to discover available identity versions when contacting https://172.31.8.10:13000/. Attempting to parse version from URL.
Could not determine a suitable URL for the plugin
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 393, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 197, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 482, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 99, in auth_ref
    return super(ClientManager, self).auth_ref
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 256, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin
    raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
DiscoveryFailure: Could not determine a suitable URL for the plugin
clean_up ListServer: Could not determine a suitable URL for the plugin
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 393, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 197, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 482, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python2.7/site-packages/openstackclient/common/clientmanager.py", line 99, in auth_ref
    return super(ClientManager, self).auth_ref
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 256, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 194, in _do_create_plugin
    raise exceptions.DiscoveryFailure('Could not determine a suitable URL '
DiscoveryFailure: Could not determine a suitable URL for the plugin

END return value: 1
Comment 14 Raildo Mascena de Sousa Filho 2020-06-29 14:23:07 UTC 
We tried to reproduce the same steps and we didn't hit the issue, also the customer is not hitting it as well. I'm closing as WORKSFORME.