Description of problem: Customer is experinecing the following traces when scaling out the overcloud: 2017-09-14 16:15:10.816 16878 DEBUG oslo_messaging._drivers.amqpdriver [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] CAST unique_id: 56b20856b0bf4ab496f9a306dbcea89c NOTIFY exchange 'heat' topic 'notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:432 2017-09-14 16:15:10.822 16878 INFO heat.engine.stack [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] Stack UPDATE FAILED (overcloud): The Referenced Attribute (Compute attributes) is incorrect. 2017-09-14 16:15:10.826 16873 DEBUG oslo_messaging._drivers.amqpdriver [req-b28dbdcf-d3a7-4170-9c0d-303f49b9f793 - - - - -] CALL msg_id: f944695bcfa24f8a9773b13fb5ef202c exchange 'heat' topic 'engine' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2017-09-14 16:15:10.828 16875 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: f944695bcfa24f8a9773b13fb5ef202c reply to reply_90eedf4db94f4a36904adcab56ba399b __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:194 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] Unhandled error in asynchronous task 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service Traceback (most recent call last): 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 134, in log_exceptions 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service gt.wait() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return self._exit_event.wait() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service current.throw(*self._exc) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service result = function(*args, **kwargs) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 125, in _start_with_trace 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return func(*args, **kwargs) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return f(*args, **kwargs) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 85, in handle_exceptions 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service {'func': func.__name__, 'msg': errmsg}) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service self.force_reraise() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service six.reraise(self.type_, self.value, self.tb) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 80, in handle_exceptions 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return func(stack, *args, **kwargs) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1187, in update 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service updater() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 170, in __call__ 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service progress_callback=progress_callback): 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 246, in as_task 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service self.start(timeout=timeout) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 192, in start 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service self.step() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 219, in step 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service poll_period = next(self._runner) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 327, in wrapper 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service subtask = next(parent) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1459, in update_task 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service rollback=action == self.ROLLBACK) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 42, in __init__ 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service for n, r in self.existing_stack.items()) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/update.py", line 42, in <genexpr> 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service for n, r in self.existing_stack.items()) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 550, in frozen_definition 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return self.t.freeze(**args) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py", line 167, in freeze 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service defn = type(self)(**dict(arg_item(a) for a in args)) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py", line 167, in <genexpr> 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service defn = type(self)(**dict(arg_item(a) for a in args)) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/rsrc_defn.py", line 159, in arg_item 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service value = function.resolve(getattr(self, attr_name)) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in resolve 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return dict((k, resolve(v)) for k, v in snippet.items()) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in <genexpr> 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return dict((k, resolve(v)) for k, v in snippet.items()) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in resolve 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return dict((k, resolve(v)) for k, v in snippet.items()) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 193, in <genexpr> 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return dict((k, resolve(v)) for k, v in snippet.items()) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/function.py", line 190, in resolve 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return snippet.result() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/hot/functions.py", line 332, in result 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return super(GetAttAllAttributes, self).result() 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/hot/functions.py", line 261, in result 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return resource.FnGetAtt(attribute, *path_components) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2063, in FnGetAtt 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service return self.get_attribute(key, *path) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 448, in get_attribute 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service self, key, False, n, *path)) for n in names) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 448, in <genexpr> 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service self, key, False, n, *path)) for n in names) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/common/grouputils.py", line 87, in get_rsrc_attr 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service resource = get_resource(stack, resource_name, use_indices, key) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/common/grouputils.py", line 83, in get_resource 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service key=key) 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service InvalidTemplateAttribute: The Referenced Attribute (Compute attributes) is incorrect. 2017-09-14 16:15:10.835 16878 ERROR heat.engine.service 2017-09-14 16:15:10.838 16878 DEBUG heat.engine.stack_lock [req-750d4fa4-7c93-4353-8aaf-14b8957be89b - - - - -] Engine 59605adc-98f6-4f65-a568-32acd65673fb released lock on stack a037e141-1d63-431a-8f2b-320b6094b84d release /usr/lib/python2.7/site-packages/heat/engine/stack_lock.py:124 2017-09-14 16:15:10.871 16875 DEBUG oslo_messaging._drivers.amqpdriver [req-bcae7afd-9a80-4685-a3c0-11f049102880 - - - - -] CALL msg_id: 7f2f2808bc51452f9221cf46e8c99216 exchange 'heat' topic 'engine' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 Based on similar bugzillas, I understand that there is an attribute with a wrong value or an unhandled attribute https://bugzilla.redhat.com/show_bug.cgi?id=1317594 https://bugzilla.redhat.com/show_bug.cgi?id=1334073 https://answers.launchpad.net/heat/+question/244184 https://bugs.launchpad.net/tripleo/+bug/1575622 Version-Release number of selected component (if applicable): OSP 10 How reproducible: always Steps to Reproduce: 1. scale the overcloud 2. 3. Actual results: the traces provided Expected results: overcloud scled out Additional info:
While it is true that InvalidTemplateAttribute normally means an error in a template causing validation to fail, in this case it's happening here: https://github.com/openstack/heat/blob/stable/newton/heat/common/grouputils.py#L82 And discussion on IRC indicates there may have been some manual modification to resources in the Heat DB - I think we need more information on that, as it may be causing the error - one reason for that raise InvalidTemplateAttribute is that heat can't load the nested resource for all items in the ResourceGroup resource - in this case the "Compute" ResourceGroup.
Hello, The resource deleted was for compute 22,23,24 because those computes don't exist in nova list. However the compute attribute error was present before deleting those heat resources. Also we have a backup of heat db we can restore to if needed.
I ran the over cloud deploy after restoring the heat database but still got the same error. RESP BODY: {"events": [{"resource_name": "overcloud", "event_time": "2017-09-15T18:42:17Z", "links": [{"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d/resources/overcloud/events/23f90348-4fdb-40e8-9df1-4fb80f48e842", "rel": "self"}, {"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d/resources/overcloud", "rel": "resource"}, {"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d", "rel": "stack"}, {"href": "http://192.168.0.1:8004/v1/230c24dced0c4cdfa9ec68555ad7557e/stacks/overcloud/a037e141-1d63-431a-8f2b-320b6094b84d", "rel": "root_stack"}], "logical_resource_id": "overcloud", "resource_status": "UPDATE_FAILED", "resource_status_reason": "The Referenced Attribute (Compute attributes) is incorrect.", "physical_resource_id": "a037e141-1d63-431a-8f2b-320b6094b84d", "id": "23f90348-4fdb-40e8-9df1-4fb80f48e842"}]}
Cancelling needinfo on assignee - we're waiting for a DB dump to be added to the support case, then we can reproduce the issue and analyse why the error is occurring.
on openstack-heat-common-10.0.1-0.20180411125639.825731d.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086