Openstack-Heat: Attempting to scale up the overcloud with more compute nodes ending with : UPDATE_FAILED . Environment: ------------ openstack-heat-templates-0.0.0-post345.el7.centos.noarch openstack-heat-api-cfn-2015.2-dev1109.el7.centos.noarch openstack-heat-common-2015.2-dev1109.el7.centos.noarch openstack-heat-api-2015.2-dev1109.el7.centos.noarch openstack-tripleo-heat-templates-0.8.4-post33.el7.centos.noarch openstack-heat-engine-2015.2-dev1109.el7.centos.noarch python-heatclient-0.5.0-post3.el7.centos.noarch openstack-heat-api-cloudwatch-2015.2-dev1109.el7.centos.noarch instack-0.0.6-post5.el7.centos.noarch instack-undercloud-0.0.0-post1712.el7.centos.noarch puppet-3.6.2-3.el7.noarch openstack-tripleo-puppet-elements-0.0.0-post56.el7.centos.noarch Steps: ------- (1) Deploy the overcloud (1 controller , 1 compute ) (2) Connect the Instack and attempt to scale up overcloud with more compute-node (by running : openstack overcloud scale stack overcloud overcloud -r $NODE_TO_SCALE_TYPE -n $NODE_SCALE_CHANGE ) Results: ---------- -----+----------------------+ 08:44:37 | id | stack_name | stack_status | creation_time | 08:44:37 +--------------------------------------+------------+---------------+----------------------+ 08:44:37 | 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z | 08:44:37 +--------------------------------------+------------+---------------+-- heat-engine.log: ----------------- 2015-05-05 14:00:47.732 16021 INFO heat.engine.resource [req-daefcd40-bf86-4390-8b4d-37e2224efc03 admin admin] Validating MultipartMime "userdata" 2015-05-05 14:00:47.732 16021 INFO heat.engine.resource [req-daefcd40-bf86-4390-8b4d-37e2224efc03 admin admin] Validating StructuredConfig "NovaComputeConfig" 2015-05-05 14:00:47.733 16021 INFO heat.engine.stack [req-daefcd40-bf86-4390-8b4d-37e2224efc03 admin admin] No content found in the "files" section for get_fi le path: file:///home/stack/tuskar_templates/hieradata/ceph.yaml 2015-05-05 14:00:47.733 16021 DEBUG oslo_messaging.rpc.dispatcher [req-daefcd40-bf86-4390-8b4d-37e2224efc03 admin admin] Expected exception during message han dling (No content found in the "files" section for get_file path: file:///home/stack/tuskar_templates/hieradata/ceph.yaml) _dispatch_and_reply /usr/lib/python 2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145 2015-05-05 14:00:47.734 16021 DEBUG oslo_messaging._drivers.amqp [req-daefcd40-bf86-4390-8b4d-37e2224efc03 admin admin] UNIQUE_ID is a65d998141014533a30bcb9e9 afddd69. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258 2015-05-05 14:00:47.736 16021 DEBUG oslo_messaging._drivers.amqp [req-daefcd40-bf86-4390-8b4d-37e2224efc03 admin admin] UNIQUE_ID is 4ccad650081d45e788a07b297 44db75b. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258 2015-05-05 14:00:47.740 16021 ERROR heat.engine.resources.stack_resource [-] update_stack 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource Traceback (most recent call last): 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", li ne 416, in update_with_template 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource args) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 259, in update_sta ck 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource args=args)) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 51, in call 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource return client.call(ctxt, method, **kwargs) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 393, in call 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource return self.prepare().call(ctxt, method, **kwargs) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource retry=self.retry) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _s end 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource timeout=timeout, retry=retry) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource retry=retry) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource raise result 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource StackValidationFailed_Remote: No content found in the "files" section for get_file pa th: file:///home/stack/tuskar_templates/hieradata/ceph.yaml 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource Traceback (most recent call last): 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 300, in wrapped 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource return func(self, ctx, *args, **kwargs) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 766, in update_stack 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource updated_stack.validate() 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource return f(*args, **kwargs) 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 629, in validate 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource raise ex 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource StackValidationFailed: No content found in the "files" section for get_file path: file:///home/stack/tuskar_templates/hieradata/ceph.yaml 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.740 16021 TRACE heat.engine.resources.stack_resource 2015-05-05 14:00:47.741 16021 INFO heat.engine.resource [-] UPDATE: TemplateResource "0" [2ac3f5a1-1e8f-433e-9fd6-14d5c7dd5e33] Stack "overcloud-Compute-ubxblx4y2ycv" [946bcc8d-d190-460b-a4c9-912cf21d1255] 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource Traceback (most recent call last): 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 489, in _action_recorder 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource yield 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 754, in update 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource args=[after, tmpl_diff, prop_diff]) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 315, in wrapper 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource step = next(subtask) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 530, in action_handler_task 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource handler_data = handler(*args) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py", line 266, in handle_update 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource self.child_params()) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 419, in updatndle_update 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource self.child_params()) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 419, in update_with_template 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource self.raise_local_exception(ex) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 309, in raise_local_exception 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource raise local_ex(message=message) 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource StackValidationFailed: No content found in the "files" section for get_file path: file:///home/stack/tuskar_templates/hieradata/ceph.yaml 2015-05-05 14:00:47.741 16021 TRACE heat.engine.resource 2015-05-05 14:00:47.767 16021 DEBUG heat.engine.scheduler [-] Task _resource_update from Stack "overcloud-Compute-ubxblx4y2ycv" [946bcc8d-d190-460b-a4c9-912cf21d1255] Update cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:265 2015-05-05 14:00:47.768 16021 DEBUG heat.engine.scheduler [-] Task _resource_update from Stack "overcloud-Compute-ubxblx4y2ycv" [946bcc8d-d190-460b-a4c9-912cf21d1255] Update cancelled cancel /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:265 2015-05-05 14:00:47.768 16021 DEBUG heat.engine.scheduler [-] Task update_task from Stack "overcloud-Compute-ubxblx4y2ycv" [946bcc8d-d190-460b-a4c9-912cf21d1255] sleeping _sleep /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:183 2015-05-05 14:00:48.302 16021 DEBUG heat.engine.scheduler [-] Task update_task from Stack "overcloud" [841dea8d-0176-4289-baa0-2a5101fd346a] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:239 2015-05-05 14:00:48.302 16021 DEBUG heat.engine.scheduler [-] Task Stack "overcloud" [841dea8d-0176-4289-baa0-2a5101fd346a] Update running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:239 2015-05-05 14:00:48.302 16021 DEBUG heat.engine.scheduler [-] Task _resource_update from Stack "overcloud" [841dea8d-0176-4289-baa0-2a5101fd346a] Update running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:239 2015-05-05 14:00:48.302 16021 DEBUG heat.engine.scheduler [-] Task _resource_update from Stack "overcloud" [841dea8d-0176-4289-baa0-2a5101fd346a] Update complete step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:245 2015-05-05 14:00:48.302 16021 DEBUG heat.engine.scheduler [-] Task _resource_update from Stack "overcloud" [841dea8d-0176-4289-baa0-2a5101fd346a] Update running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:239 2015-05-05 14:00:48.331 16021 INFO heat.engine.environment [-] Registering OS::TripleO::Controller::Net::SoftwareConfig -> file:///home/stack/tuskar_templates/net-config-bridge.yaml 2015-05-05 14:00:48.332 16021 INFO heat.engine.environment [-] Registering OS::TripleO::CephClusterConfig::SoftwareConfig -> file:///home/stack/tuskar_templates/puppet/ceph-cluster-config.yaml 2015-05-05 14:00:48.332 16021 INFO heat.engine.environment [-] Registering OS::TripleO::ObjectStorage -> file:///home/stack/tuskar_templates/puppet/swift-storage-puppet.yaml 2015-05-05 14:00:48.332 16021 INFO heat.engine.environment [-] Registering Tuskar::Compute-1 -> file:///home/stack/tuskar_templates/provider-Compute-1.yaml 2015-05-05 14:00:48.332 16021 INFO heat.engine.environment [-] Registering OS::TripleO::NodeExtraConfigPost -> file:///home/stack/tuskar_templates/extraconfig/post_deploy/default.yaml 2015-05-05 14:00:48.332 16021 INFO heat.engine.environment [-] Registering OS::TripleO::NodeUserData -> file:///home/stack/tuskar_templates/firstboot/userdata console view: -------------- 08:20:10 cmd: 08:20:10 source /home/stack/stackrc; source /home/stack/scale-setup-env; openstack overcloud scale stack overcloud overcloud -r $NODE_TO_SCALE_TYPE -n $NODE_SCALE_CHANGE 08:20:10 08:20:10 start: 08:20:10 2015-05-05 12:13:55.396031 08:20:10 08:20:10 end: 08:20:10 2015-05-05 12:13:59.991489 08:20:10 08:20:10 delta: 08:20:10 0:00:04.595458 08:20:10 08:20:10 stdout: 08:20:10 08:20:10 08:20:10 stderr: 08:20:10 08:20:10 08:20:10 TASK: [rdo-manager/scale-nodes | poll for heat stack-list to go to COMPLETE] *** 08:20:10 [[ previous task time: 0:00:06.966236 = 6.97s / 16.93s ]] 08:20:10 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:20:10 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:20:10 Result from run 1 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:17:07.884222', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 1, u'start': u'2015-05-05 12:17:05.710572', u'delta': u'0:00:02.173650', u'stderr': u'', u'rc': 0, u'warnings': []} 08:20:10 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:35:26 Result from run 2 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:20:11.100551', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 2, u'start': u'2015-05-05 12:20:10.437438', u'delta': u'0:00:00.663113', u'stderr': u'', u'rc': 0, u'warnings': []} 08:35:26 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:35:26 Result from run 3 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:23:14.371062', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 3, u'start': u'2015-05-05 12:23:13.720026', u'delta': u'0:00:00.651036', u'stderr': u'', u'rc': 0, u'warnings': []} 08:35:26 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:35:26 Result from run 4 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:26:17.952249', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 4, u'start': u'2015-05-05 12:26:17.294525', u'delta': u'0:00:00.657724', u'stderr': u'', u'rc': 0, u'warnings': []} 08:35:26 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:35:26 Result from run 5 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:29:21.005924', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 5, u'start': u'2015-05-05 12:29:20.384882', u'delta': u'0:00:00.621042', u'stderr': u'', u'rc': 0, u'warnings': []} 08:35:26 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:35:26 Result from run 6 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:32:24.104918', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 6, u'start': u'2015-05-05 12:32:23.452147', u'delta': u'0:00:00.652771', u'stderr': u'', u'rc': 0, u'warnings': []} 08:35:26 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:44:37 Result from run 7 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:35:27.171849', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 7, u'start': u'2015-05-05 12:35:26.489077', u'delta': u'0:00:00.682772', u'stderr': u'', u'rc': 0, u'warnings': []} 08:44:37 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:44:37 Result from run 8 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:38:30.685211', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 8, u'start': u'2015-05-05 12:38:30.088087', u'delta': u'0:00:00.597124', u'stderr': u'', u'rc': 0, u'warnings': []} 08:44:37 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:44:37 Result from run 9 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:41:34.056891', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 9, u'start': u'2015-05-05 12:41:33.365145', u'delta': u'0:00:00.691746', u'stderr': u'', u'rc': 0, u'warnings': []} 08:44:37 <undercloud> REMOTE_MODULE command source /home/stack/stackrc; heat stack-list #USE_SHELL 08:44:37 Result from run 10 is: {u'cmd': u'source /home/stack/stackrc; heat stack-list', u'end': u'2015-05-05 12:44:37.924159', u'stdout': u'+--------------------------------------+------------+---------------+----------------------+\n| id | stack_name | stack_status | creation_time |\n+--------------------------------------+------------+---------------+----------------------+\n| 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z |\n+--------------------------------------+------------+---------------+----------------------+', u'changed': True, 'attempts': 10, u'start': u'2015-05-05 12:44:37.074921', u'delta': u'0:00:00.849238', u'stderr': u'', u'rc': 0, u'warnings': []} 08:44:37 failed: [groups=undercloud] => {"attempts": 10, "changed": true, "cmd": "source /home/stack/stackrc; heat stack-list", "delta": "0:00:00.849238", "end": "2015-05-05 12:44:37.924159", "failed": true, "rc": 0, "start": "2015-05-05 12:44:37.074921", "warnings": []} 08:44:37 stdout: +--------------------------------------+------------+---------------+----------------------+ 08:44:37 | id | stack_name | stack_status | creation_time | 08:44:37 +--------------------------------------+------------+---------------+----------------------+ 08:44:37 | 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z | 08:44:37 +--------------------------------------+------------+---------------+----------------------+ 08:44:37 msg: Task failed as maximum retries was encountered 08:44:37 08:44:37 cmd: 08:44:37 source /home/stack/stackrc; heat stack-list 08:44:37 08:44:37 start: 08:44:37 2015-05-05 12:44:37.074921 08:44:37 08:44:37 end: 08:44:37 2015-05-05 12:44:37.924159 08:44:37 08:44:37 delta: 08:44:37 0:00:00.849238 08:44:37 08:44:37 msg: 08:44:37 Task failed as maximum retries was encountered 08:44:37 08:44:37 stdout: 08:44:37 +--------------------------------------+------------+---------------+----------------------+ 08:44:37 | id | stack_name | stack_status | creation_time | 08:44:37 +--------------------------------------+------------+---------------+----------------------+ 08:44:37 | 01413073-ea4a-40cf-8f73-1566ae5af2f6 | overcloud | UPDATE_FAILED | 2015-05-05T07:47:22Z | 08:44:37 +--------------------------------------+------------+---------------+----------------------+ 08:44:37 08:44:37 stderr: 08:44:37 08:44:37 08:44:37 FATAL: all hosts have already failed -- aborting 08:44:37 08:44:37 PLAY RECAP ******************************************************************** 08:44:37 to retry, use: --limit @/home/rhos-ci/rdo-manager-scale-overcloud.retry 08:44:37 08:44:37 groups=undercloud : ok=5 changed=1 unreachable=0 failed=1 08:44:37 08:44:37 [[ previous task time: 0:30:37.863044 = 1837.86s / 1854.79s ]] 08:44:37 [[ previous play time: 0:30:54.529968 = 1854.53s / 1854.79s ]] 08:44:37 [[ previous playbook time: 0:30:54.790062 = 1854.79s / 1854.79s ]]
So the exception at least looks very similar to the one in bug 1215511, but in this case you're definitely using python-heatclient 0.5.0. And I assume you also have the fix for bug 1212625, because that has been in the midstream for a while. So there must be something else going on :/
Database spelunking confirms that this is *not* the same as bug 1215511 - the templates for both the regular and backup stacks contain the correct key in the files section. Inspection of the code also confirms that the fix for bug 1212625 is indeed present.
Steve Baker pointed out that this could be caused by bug 1212740.
Before the tripleo-common ScaleManager calls stacks.update it needs to prepare the environment and files by calling template_utils.get_template_contents and template_utils.process_multiple_environments_and_files http://git.openstack.org/cgit/openstack/python-heatclient/tree/heatclient/v1/shell.py#n449 Otherwise files will be missing in the request, and file paths will not be normalised.
As a (nasty) workaround for BZ 1212740 scale-out code does filename replacement before sending data to heat. Albeit this is not nice, this worked with older rpms last week. I will submit a simple patch which saves temporarily saves template files so get_template_contents can be used to process/prepare them, but I'm not sure this regression is caused by this: If I do a simple test: 1) instack-deploy-overcloud --tuskar 2) cp -a /home/stach/tuskar_templates /home/stach/tuskar_templates.2 3) heat stack-update -f tuskar_templates.2/plan.yaml -e tuskar_templates.2/environment.yaml overcloud Then stack update fails anyway, in stack events I can see: | Controller | 2dc68516-f3e9-4757-bb2d-04e9d5080dba | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "StackValidationFailed: Property error : : resources.ControllerConfig.properties.config: : No content found in the "files" section for get_file path: file:///home/stack/tuskar_template | UPDATE_FAILED | 2015-05-06T07:14:58Z | | Compute | 91d3476d-68b7-4394-b48d-08791ff140e1 | ResourceUnknownStatus: Resource failed - Unknown status FAILED due to "StackValidationFailed: No content found in the "files" section for get_file path: file:///home/stack/tuskar_templates/hieradata/ceph.yaml" | UPDATE_FAILED | 2015-05-06T07:15:02Z |
I've updated the scale out code to use get_template_contents and process_multiple_environments_and_files to make sure the problem is not on client side and I get same error as in Comment #7.
So it looks like bug 1212625 is not fixed, even with the patch in place. It's again looking for the old path (tuskar_templates, not tuskar_templates.2) and adding extra debug shows that it's doing so from a files collection where everything is named with the new path.
OK, we tracked down the problem. In this case, it was that we were passing an old version of a template (with old normalised paths) to stack update for a template resource (which the members of the ResourceGroups are). Basically TemplateResource failed to account for the possibility that the template name had changed in the environment on an update, and the bug was further masked by the fact that it falls back to just using the current template when it really shouldn't. So this is completely independent of bug 1212625, despite having almost identical symptoms, and fixes for both are required.
Fix posted upstream: https://review.openstack.org/#/c/180843/
Backport to Kilo proposed upstream and cherry-picked into rdo-management mgt-kilo branch.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2015:1548