Bug 1492120 - InvalidTemplateAttribute: The Referenced Attribute (Compute attributes) is incorrect while scaling out the overcloud
Summary: InvalidTemplateAttribute: The Referenced Attribute (Compute attributes) is in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
urgent
Target Milestone: Upstream M1
: 13.0 (Queens)
Assignee: Rabi Mishra
QA Contact: Ronnie Rasouli
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-15 14:05 UTC by Eduard Barrera
Modified: 2019-07-31 21:10 UTC (History)
17 users (show)

Fixed In Version: openstack-heat-10.0.1-0.20180314232329.c2a66b1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:36:16 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 511735 None MERGED Raise NotFound() when group member does not exist 2020-08-01 22:39:07 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:37:03 UTC

Description Eduard Barrera 2017-09-15 14:05:05 UTC
Description of problem:

Customer is experinecing the following traces when scaling out the overcloud:

2017-09-14 16:15:10.816 16878 DEBUG oslo_messaging._drivers.amqpdriver [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] CAST unique_id: 56b20856b0bf4ab496f9a306dbcea89c NOTIFY exchange 'heat' topic 'notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:432
2017-09-14 16:15:10.822 16878 INFO heat.engine.stack [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] Stack UPDATE FAILED (overcloud): The Referenced Attribute (Compute attributes) is incorrect.
2017-09-14 16:15:10.826 16873 DEBUG oslo_messaging._drivers.amqpdriver [req-b28dbdcf-d3a7-4170-9c0d-303f49b9f793 - - - - -] CALL msg_id: f944695bcfa24f8a9773b13fb5ef202c exchange 'heat' topic 'engine' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
2017-09-14 16:15:10.828 16875 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: f944695bcfa24f8a9773b13fb5ef202c reply to reply_90eedf4db94f4a36904adcab56ba399b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] Unhandled error in asynchronous task
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service Traceback (most recent call last):
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 134, in log_exceptions
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     gt.wait()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return self._exit_event.wait()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     current.throw(*self._exc)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     result = function(*args, **kwargs)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 125, in _start_with_trace
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return func(*args, **kwargs)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return f(*args, **kwargs)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 85, in handle_exceptions
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     {'func': func.__name__, 'msg': errmsg})
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     self.force_reraise()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     six.reraise(self.type_, self.value, self.tb)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 80, in handle_exceptions
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return func(stack, *args, **kwargs)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1187, in update
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     updater()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 170, in __call__
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     progress_callback=progress_callback):
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 246, in as_task
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     self.start(timeout=timeout)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 192, in start
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     self.step()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 219, in step
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     poll_period = next(self._runner)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 327, in wrapper
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     subtask = next(parent)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1459, in update_task
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     rollback=action == self.ROLLBACK)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 42, in __init__
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     for n, r in self.existing_stack.items())
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 42, in <genexpr>
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     for n, r in self.existing_stack.items())
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 550, in frozen_definition
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return self.t.freeze(**args)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py", line 167, in freeze
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     defn = type(self)(**dict(arg_item(a) for a in args))
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py", line 167, in <genexpr>
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     defn = type(self)(**dict(arg_item(a) for a in args))
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py", line 159, in arg_item
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     value = function.resolve(getattr(self, attr_name))
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in resolve
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return dict((k, resolve(v)) for k, v in snippet.items())
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in <genexpr>
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return dict((k, resolve(v)) for k, v in snippet.items())
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in resolve
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return dict((k, resolve(v)) for k, v in snippet.items())
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in <genexpr>
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return dict((k, resolve(v)) for k, v in snippet.items())
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 190, in resolve
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return snippet.result()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/hot/functions.py", line 332, in result
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return super(GetAttAllAttributes, self).result()
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/hot/functions.py", line 261, in result
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return resource.FnGetAtt(attribute, *path_components)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2063, in FnGetAtt
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     return self.get_attribute(key, *path)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 448, in get_attribute
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     self, key, False, n, *path)) for n in names)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 448, in <genexpr>
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     self, key, False, n, *path)) for n in names)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/common/grouputils.py", line 87, in get_rsrc_attr
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     resource = get_resource(stack, resource_name, use_indices, key)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service   File "/usr/lib/python2.7/site-packages/heat/common/grouputils.py", line 83, in get_resource
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service     key=key)
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service InvalidTemplateAttribute: The Referenced Attribute (Compute attributes) is incorrect.
2017-09-14 16:15:10.835 16878 ERROR heat.engine.service
2017-09-14 16:15:10.838 16878 DEBUG heat.engine.stack_lock [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] Engine 59605adc-98f6-4f65-a568-32acd65673fb released lock on stack a037e141-1d63-431a-8f2b-320b6094b84d release /usr/lib/python2.7/site-packages/heat/engine/stack_lock.py:124
2017-09-14 16:15:10.871 16875 DEBUG oslo_messaging._drivers.amqpdriver [req-bcae7afd-9a80-4685-a3c0-11f049102880 - - - - -] CALL msg_id: 7f2f2808bc51452f9221cf46e8c99216 exchange 'heat' topic 'engine' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448


Based on similar bugzillas, I understand that there is an attribute with a wrong value or an unhandled attribute

https://bugzilla.redhat.com/show_bug.cgi?id=1317594
https://bugzilla.redhat.com/show_bug.cgi?id=1334073
https://answers.launchpad.net/heat/+question/244184
https://bugs.launchpad.net/tripleo/+bug/1575622



Version-Release number of selected component (if applicable):
OSP 10

How reproducible:
always

Steps to Reproduce:
1. scale the overcloud
2.
3.

Actual results:
the traces provided

Expected results:
overcloud scled out

Additional info:

Comment 4 Steven Hardy 2017-09-15 15:36:56 UTC
While it is true that InvalidTemplateAttribute normally means an error in a template causing validation to fail, in this case it's happening here:

https://github.com/openstack/heat/blob/stable/newton/heat/common/grouputils.py#L82

And discussion on IRC indicates there may have been some manual modification to resources in the Heat DB - I think we need more information on that, as it may be causing the error - one reason for that raise InvalidTemplateAttribute is that heat can't load the nested resource for all items in the ResourceGroup resource - in this case the "Compute" ResourceGroup.

Comment 5 Jeremy 2017-09-15 15:45:10 UTC
Hello,
The resource deleted was for compute 22,23,24 because those computes don't exist in nova list. However the compute attribute error was present before deleting those heat resources. Also we have a backup of heat db we can restore to if needed.

Comment 6 Jeremy 2017-09-15 18:53:44 UTC
I ran the over cloud deploy after restoring the heat database but still got the same error.

RESP BODY: {"events": [{"resource_name": "overcloud", "event_time": "2017-09-15T18:42:17Z", "links": [{"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d/resources/overcloud/events/23f90348-4fdb-40e8-9df1-4fb80f48e842", "rel": "self"}, {"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d/resources/overcloud", "rel": "resource"}, {"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d", "rel": "stack"}, {"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d", "rel": "root_stack"}], "logical_resource_id": "overcloud", "resource_status": "UPDATE_FAILED", "resource_status_reason": "The Referenced Attribute (Compute attributes) is incorrect.", "physical_resource_id": "a037e141-1d63-431a-8f2b-320b6094b84d", "id": "23f90348-4fdb-40e8-9df1-4fb80f48e842"}]}

Comment 9 Steven Hardy 2017-09-17 15:05:23 UTC
Cancelling needinfo on assignee - we're waiting for a DB dump to be added to the support case, then we can reproduce the issue and analyse why the error is occurring.

Comment 22 Ronnie Rasouli 2018-05-23 08:15:38 UTC
on openstack-heat-common-10.0.1-0.20180411125639.825731d.el7ost.noarch

Comment 24 errata-xmlrpc 2018-06-27 13:36:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.