Bug 1709846
| Summary: | OVN upgrade from osp13 to osp14 failed on Overcloud upgrade preparation step | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eran Kuris <ekuris> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Jose Luis Franco <jfrancoa> |
| Status: | CLOSED ERRATA | QA Contact: | Sasha Smolyak <ssmolyak> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 14.0 (Rocky) | CC: | abishop, aschultz, dbecker, jfrancoa, lbezdick, mburns, morazi, rrasouli |
| Target Milestone: | z3 | Keywords: | Triaged, ZStream |
| Target Release: | 14.0 (Rocky) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-9.3.1-0.20190513171736.9995be9.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-02 20:08:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Might https://bugs.launchpad.net/tripleo/+bug/1759326 related? Hello Eran, would it be possible to login into some environment with the issue reproduced? From the logs I really can't see what's the reason why that service wouldn't be found...the default resource_registry is setting it to Noop (Heat::None), so it should just be skipped...and I can't find any service name missmatching... Thanks. (In reply to Jose Luis Franco from comment #4) > Hello Eran, > > would it be possible to login into some environment with the issue > reproduced? From the logs I really can't see what's the reason why that > service wouldn't be found...the default resource_registry is setting it to > Noop (Heat::None), so it should just be skipped...and I can't find any > service name missmatching... > > Thanks. Luis, I don't have the setup but I will try to reproduce it on one of my hosts and I will post the details. CinderBackendPure is an optional service associated with the cinder driver associated with the Pure Storage SAN. Upstream patches merged on stable/rocky [1] and stable/queens [2] on March 18. I think what may have happened (I'd need to see fresh deployment logs) is the OSP-13 build contained the patch that added support for the service, but the version of OSP-14 is _not_ contain the corresponding patch. I think the problem will not occur as long as both OSP-13 and 14 contain the patch that adds support for CinderBackendPure. [1] https://review.opendev.org/641764 [2] https://review.opendev.org/641766 The patch seems to be include in the rhos-14.0-patches already, probably part of some periodic import. The issue was fixed and verified: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-upgrade-13-14_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/102/ core_puddle: 2019-05-15.1 (osp13) core_puddle: 2019-05-31.1 (osp14) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672 |
Description of problem: The upgrade process failed from OSP13OVN to OSP14OVN on overcloud upgrade preparation step 2019-05-07 14:17:46 | {u'result': u'Failure caused by error in tasks: set_get_tempurl\n\n set_get_tempurl [task_ex_id=68e6b02f-08ca-4e6d-9da7-dbaf66932054] -> Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1299, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1083, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 316, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nStackValidationFailed: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\n\n [action_ex_id=7a94c1b0-6ce1-4f8b-8e28-3f8d114e0833, idx=0]: Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1299, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1083, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 316, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nStackValidationFailed: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\n\n', u'swift_container': u'overcloud-swift-rings'} 2019-05-07 14:17:46 | 2019-05-07 14:17:46.024 77085 ERROR openstack [-] Exception updating plan: {u'result': u'Failure caused by error in tasks: set_get_tempurl\n\n set_get_tempurl [task_ex_id=68e6b02f-08ca-4e6d-9da7-dbaf66932054] -> Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1299, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1083, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 316, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nStackValidationFailed: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\n\n [action_ex_id=7a94c1b0-6ce1-4f8b-8e28-3f8d114e0833, idx=0]: Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1299, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1083, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 316, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nStackValidationFailed: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\n\n', u'swift_container': u'overcloud-swift-rings'}: WorkflowServiceError: Exception updating plan: {u'result': u'Failure caused by error in tasks: set_get_tempurl\n\n set_get_tempurl [task_ex_id=68e6b02f-08ca-4e6d-9da7-dbaf66932054] -> Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1299, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1083, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 316, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nStackValidationFailed: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\n\n [action_ex_id=7a94c1b0-6ce1-4f8b-8e28-3f8d114e0833, idx=0]: Error validating environment for plan overcloud: ERROR: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\nTraceback (most recent call last):\n\n File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 409, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1299, in validate_template\n result.update(stack.get_nested_parameters(filter_parameter))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1101, in get_nested_parameters\n params.update(nested.get_nested_parameters(filter_func))\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1083, in get_nested_parameters\n for name, rsrc in six.iteritems(self.resources):\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 316, in resources\n self._resources = self._resources_for_defn(self.defn)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in _resources_for_defn\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 310, in <dictcomp>\n for name in stack_defn.enabled_rsrc_names()\n\n File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 174, in __new__\n resource_name=name)\n\n File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 561, in get_class_to_instantiate\n raise exception.StackValidationFailed(message=six.text_type(exc))\n\nStackValidationFailed: The Resource Type (OS::TripleO::Services::CinderBackendPure) could not be found.\n\n', u'swift_container': u'overcloud-swift-rings'}[00m Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. run CI upgrade job https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-upgrade-13-14_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/93/artifact/.sh/ir-tripleo-overcloud-upgrade.log 2. 3. Actual results: Expected results: Additional info: