Bug 1474749
Summary: | Openstack Upgrade to 11 is failing, need to find the root casue | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Eduard Barrera <ebarrera> |
Component: | openstack-tripleo | Assignee: | Sofer Athlan-Guyot <sathlang> |
Status: | CLOSED NOTABUG | QA Contact: | Amit Ugol <augol> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 10.0 (Newton) | CC: | ebarrera, jthomas, mburns, mschuppe, ramishra, rhel-osp-director-maint, sathlang, therve, zbitter |
Target Milestone: | --- | Keywords: | Triaged |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2017-08-18 11:58:42 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Eduard Barrera
2017-07-25 10:26:19 UTC
Created attachment 1304171 [details]
last errors the customer got
Here's the traceback: RESP BODY: {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "The specified reference \"NetworkerAllNodesValidationDeployment\" (in AllNodesExtraConfig) is incorrect.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 407, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 2001, in list_stack_resources\n for resource in rsrcs]\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/api.py\", line 345, in format_stack_resource\n rpc_api.RES_REQUIRED_BY: resource.required_by(),\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/resource.py\", line 665, in required_by\n return [r.name for r in self.stack.dependencies.required_by(self)]\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/stack.py\", line 403, in dependencies\n ignore_errors=self.id is not None)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/stack.py\", line 477, in _get_dependencies\n res.add_explicit_dependencies(deps)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/resource.py\", line 639, in add_explicit_dependencies\n for dep in self.t.dependencies(self.stack):\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py\", line 238, in dependencies\n filter(None, (get_resource(dep) for dep in explicit_depends)),\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py\", line 238, in <genexpr>\n filter(None, (get_resource(dep) for dep in explicit_depends)),\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py\", line 215, in get_resource\n key=self.name)\n\nInvalidTemplateReference: The specified reference \"NetworkerAllNodesValidationDeployment\" (in AllNodesExtraConfig) is incorrect.\n", "type": "InvalidTemplateReference"}, "title": "Bad Request"} So it's failing the calculate the dependency graph (that's bad - without it you can't do an update... this is not something that's ever supposed to get out of sync). Looking at the templates: AllNodesExtraConfig: type: OS::TripleO::AllNodesExtraConfig depends_on: - UpdateWorkflow {% for role in roles %} - {{role.name}}AllNodesValidationDeployment {% endfor %} This is the depends_on we're failing to resolve. Somehow the template got stored with depends_on NetworkerAllNodesValidationDeployment here in the AllNodesExtraConfig resource, but without actually having a NetworkerAllNodesValidationDeployment resource in the template. This shouldn't be possible even if an update fails part-way through: because of the dependency relationship if a role is added then the {{role.name}}AllNodesValidationDeployment should be copied into the template before the new AllNodesExtraConfig definition (with the new depends_on) is copied into the template, and vice-versa if a role is deleted. There's no heat-engine log in the sosreport, so not much information we can go on to figure out what happened. Created attachment 1308006 [details]
new errors
|