Description of problem: ----------------------- Preparing of oc upgrade playbooks fails: openstack overcloud upgrade --init-update \ --container-registry-file images.yaml \ -e disabled-services.yml Waiting for messages on queue 'tripleo' with no timeout. Started Mistral Workflow tripleo.package_update.v1.package_update_plan. Execution ID: 31f39151-b7a7-472c-b472-0712a8e5d8d4 2018-02-01 11:07:17Z [ServiceNetMap]: UPDATE_IN_PROGRESS state changed 2018-02-01 11:07:18Z [overcloud-ServiceNetMap-nx7ak2zxtabg]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-02-01 11:07:18Z [overcloud-ServiceNetMap-nx7ak2zxtabg]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-02-01 11:07:19Z [Networks]: UPDATE_IN_PROGRESS state changed 2018-02-01 11:07:19Z [ServiceNetMap]: UPDATE_COMPLETE state changed ... 2018-02-01 11:07:42Z [overcloud-VipMap-bywnffs4eklz]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-02-01 11:07:43Z [VipMap]: UPDATE_COMPLETE state changed 2018-02-01 11:07:43Z [EndpointMap]: UPDATE_IN_PROGRESS state changed 2018-02-01 11:07:45Z [EndpointMap]: UPDATE_COMPLETE state changed 2018-02-01 11:07:45Z [overcloud]: UPDATE_FAILED Error in EndpointMap output endpoint_map: Invalid URL port "" for make_url called with {u'host': u'', u'scheme': '', u'port': ''} Stack overcloud UPDATE_FAILED From heat-engine.log: --------------------- Stack "overcloud" [fd4e59de-fabd-4f37-862a-3663563a129a] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:209 2018-02-01 05:32:19.670 2981 DEBUG heat.engine.scheduler [req-fdb86d94-2251-4cbf-9d18-328041ee4810 - admin - default default] Task update from TemplateResource "EndpointMap" [bd0dc0f1-7ca3-4383-8738-50cfa760986f ] Stack "overcloud" [fd4e59de-fabd-4f37-862a-3663563a129a] complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:215 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource [req-fdb86d94-2251-4cbf-9d18-328041ee4810 - admin - default default] Unexpected exception in resource check.: TemplateOutputError: Error in EndpointM ap output endpoint_map: Invalid URL port "" for make_url called with {u'host': u'', u'scheme': '', u'port': ''} 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource Traceback (most recent call last): 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 321, in check 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource rsrc, stack) 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 253, in _initiate_propagate_resource 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource input_data = _get_input_data(req_node, input_forward_data) 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 236, in _get_input_data 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource return rsrc.node_data().as_dict() 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource File "/usr/lib/python2.7/site-packages/heat/engine/node_data.py", line 85, in as_dict 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource raise v 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource TemplateOutputError: Error in EndpointMap output endpoint_map: Invalid URL port "" for make_url called with {u'host': u'', u'scheme': '', u'port': '' } 2018-02-01 05:32:19.885 2981 ERROR heat.engine.check_resource 2018-02-01 05:32:19.887 2981 INFO heat.engine.stack [req-fdb86d94-2251-4cbf-9d18-328041ee4810 - admin - default default] Stack UPDATE FAILED (overcloud): Error in EndpointMap output endpoint_map: Invalid URL por t "" for make_url called with {u'host': u'', u'scheme': '', u'port': ''} 2018-02-01 05:32:19.898 2981 DEBUG heat.engine.stack [req-fdb86d94-2251-4cbf-9d18-328041ee4810 - admin - default default] Persisting stack overcloud status UPDATE FAILED _send_notification_and_add_event /usr/lib /python2.7/site-packages/heat/engine/stack.py:1019 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker [req-fdb86d94-2251-4cbf-9d18-328041ee4810 - admin - default default] Unhandled exception in check_resource: TemplateOutputError: Error in EndpointMap output endpoint_map: Invalid URL port "" for make_url called with {u'host': u'', u'scheme': '', u'port': ''} 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker Traceback (most recent call last): 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/worker.py", line 45, in exception_wrapper 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker return func(*args, **kwargs) 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/worker.py", line 196, in check_resource 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker is_update, adopt_stack_data, rsrc, stack) 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 327, in check 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker stack, msg) 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker self.force_reraise() 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker six.reraise(self.type_, self.value, self.tb) 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 321, in check 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker rsrc, stack) 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 253, in _initiate_propagate_resource 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker input_data = _get_input_data(req_node, input_forward_data) 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/check_resource.py", line 236, in _get_input_data 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker return rsrc.node_data().as_dict() 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker File "/usr/lib/python2.7/site-packages/heat/engine/node_data.py", line 85, in as_dict 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker raise v 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker TemplateOutputError: Error in EndpointMap output endpoint_map: Invalid URL port "" for make_url called with {u'host': u'', u'scheme': '', u'port': ''} 2018-02-01 05:32:19.932 2981 ERROR heat.engine.worker 2018-02-01 05:32:22.416 2982 INFO heat.engine.resource [req-c97c025e-415a-4b5c-a884-0cd0088eb922 - - - - -] The Referenced Attribute (0 blacklist_ip_address) is incorrect. Version-Release number of selected component (if applicable): ------------------------------------------------------------- python-heat-agent-1.5.2-0.20180112100314.9f9ae41.el7ost.noarch python2-heatclient-1.13.0-0.20171215101017.fbff3c6.el7ost.noarch heat-cfntools-1.3.0-2.el7ost.noarch openstack-heat-common-10.0.0-0.20180123134729.85a3e98.el7ost.noarch openstack-tripleo-heat-templates-8.0.0-0.20180122224016.el7ost.noarch openstack-heat-api-cfn-10.0.0-0.20180123134729.85a3e98.el7ost.noarch openstack-heat-api-10.0.0-0.20180123134729.85a3e98.el7ost.noarch puppet-heat-12.2.0-0.20180119235231.7870a0f.el7ost.noarch openstack-heat-engine-10.0.0-0.20180123134729.85a3e98.el7ost.noarch python-tripleoclient-9.0.1-0.20180119233147.el7ost.noarch Steps to Reproduce: ------------------- 1. Upgrade RHOS-12 uc (repo 2018-01-26.3) 2. Apply patches 526213 and 526214 3. Try running uc upgrade init Actual results: --------------- Stack update fail Expected results: ----------------- Stack update succeeds
Investigation showed the problem with patches that are not yet merged. DockerRegistryInternal was introduced in RHOS13. Current upgrade process is trying to update Heat resources without updating Heat templates. So, the default values for DockerRegistryInternal are not set in template.
*** Bug 1553619 has been marked as a duplicate of this bug. ***
The investigation showed that SSL templates were not updated properly. The issue is related to environments with enabled SSL. The overcloud was deployed as openstack overcloud deploy \ ... \ -e /home/stack/enable-tls.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \ ... As a workaround 'openstack overcloud upgrade prepare' should be modified as openstack overcloud upgrade prepare ... \ -e ~/tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ ... That will allow heat to merge templates correctly. It looks like tripleoclient/v1/undercloud_config.py doesn't merge ~/tripleo-heat-templates/environments/ssl correctly
Heat needs the updated EndpointMap which was originally overridden by tls-endpoints-public-ip.yaml. If you don't pass it to the stack update, it has no way of knowing that the file on disk should be taken. It might be a specific case because overriding EndpointMap is tricky, but the issue may appear with other env files.
Thank you Thomas. We decided, for safety reasons, to provide all the env files for the prepare command. Looks like the endpoint_map is the only 'issue' for now, but we can't predict other specific cases like this. This will turn to a doc patch then.
I'm closing this one, and I opened instead, for Documentation: #1568690 *** This bug has been marked as a duplicate of bug 1568690 ***