+++ This bug was initially created as a clone of Bug #1320454 +++
Description of problem:
This is a mixed version problem. It happens when you do a full osp7 install (undercloud and overcloud) then you just upgrade the undercloud to osp8. After you do that things work just fine, you can create and delete instances on the overcloud and so on, but if you try to run any scale or just a simple template via tht it fails and everything gets borked on a UPDATE_FAIL status
Version-Release number of selected component (if applicable):
How reproducible:
Very
Steps to Reproduce:
1. do a full osp7 install
2. upgrade the undercloud as per the kbase document
3. try run any template openstack overcloud deploy --templates ~/local_templates -e foo.yaml
[stack@undercloud ~]$cat foo.yaml
resource_registry:
OS::TripleO::NodeExtraConfig: install_nmap_template.yaml
[stack@undercloud ~]$ cat install_nmap_template.yaml
heat_template_version: 2014-10-16
description: 'Install Nmap'
parameters:
server:
type: string
resources:
NmapConfig:
type: OS::Heat::SoftwareConfig
properties:
group: script
config: |
#!/bin/sh
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
yum -y install nmap
NmapDeployment:
type: OS::Heat::SoftwareDeployment
properties:
name: NmapDeployment
config: {get_resource: NmapConfig}
server: {get_param: server}
outputs:
deploy_stdout:
value: "None"
Actual results:
It fails
Expected results:
it should pass
Additional info:
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack [req-a85dddaa-68eb-47b0-89f8-92e066e6e4d4 9b9513cbe51c442fb8bac58d251133cc d73003a5c802451693a4e2e12ab91323] Exception: "u'1:1000'" is not a list
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack Traceback (most recent call last):
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 674, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack result = res.validate()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py", line 279, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack return super(TemplateResource, self).validate()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 57, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack super(StackResource, self).validate()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1135, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack return self.validate_template()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1152, in validate_template
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack with_value=self.stack.strict_validate)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 383, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack self._get_property_value(key, validate=True)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 451, in _get_property_value
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack return prop.get_value(None, validate)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 326, in get_value
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack _value = self._get_list(value, validate)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 296, in _get_list
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack raise TypeError(_('"%s" is not a list') % repr(value))
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack TypeError: "u'1:1000'" is not a list
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource [-] update_stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource Traceback (most recent call last):
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 434, in update_with_template
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource args)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 269, in update_stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource args=args))
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 61, in call
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource return client.call(ctxt, method, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 403, in call
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource return self.prepare().call(ctxt, method, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource retry=self.retry)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource timeout=timeout, retry=retry)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource retry=retry)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource raise result
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource StackValidationFailed_Remote: "u'1:1000'" is not a list
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource Traceback (most recent call last):
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 308, in wrapped
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource return func(self, ctx, *args, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 868, in update_stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource cnxt, current_stack, template, params, files, args)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 830, in _prepare_stack_updates
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource updated_stack.validate()
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource return f(*args, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 685, in validate
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource message=encodeutils.safe_decode(six.text_type(ex)))
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource StackValidationFailed: "u'1:1000'" is not a list
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource yield
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource prop_diff])
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource step = next(subtask)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource handler_data = handler(*args)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 427, in handle_update
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource checkers[0].start()
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource self.step()
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource next(self._runner)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource timeout)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource self.raise_local_exception(ex)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource raise exception.ResourceFailure(message, self, action=self.action)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource ResourceFailure: resources.Compute: "u'1:1000'" is not a list
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource
--- Additional comment from Adriano Petrich on 2016-03-23 06:49:13 EDT ---
Just for completeness
this bug looks a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1303084
but it is not the same.
I verified and the fixes for that bug were applied on my env such as this one:
https://review.openstack.org/#/c/286874/1/heat/engine/resource.py
--- Additional comment from Dan Yasny on 2016-03-23 10:07:22 EDT ---
Reproduced again in manual testing
--- Additional comment from marios on 2016-03-24 08:54:55 EDT ---
wasn't able to reproduce this; how long before this failure happens (ish, I mean is it immediate?). Here is what I did ftr (in particular, are you sure you included all the environment files for the stack update?):
deploy HA net-iso overcloud like:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server "0.fedora.pool.ntp.org"
update packages on the undercloud and rerun openstack undercloud install.
I used your example templates, (but just echo "FOOOO from `hostname`" rather than install nmap):
openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e foo.yaml
This ran (almost) to completion, I hit an error but I think it is because I restarted openstack-nova-api mid-way through the update (forgot to after the undercloud update). In particular I didn't see the
"u'1:1000'" is not a list and on the compute/controller I see my 'config' has run OK
[root@overcloud-controller-0 ~]# journalctl -fn 1000000 | grep "FOOOO"
Mar 24 11:51:57 overcloud-controller-0.localdomain os-collect-config[5053]: [2016-03-24 11:51:57,215] (heat-config) [INFO] {"deploy_stdout": "FOOOO from overcloud-controller-0.localdomain\n", "deploy_stderr": "", "deploy_status_code": 0}
[root@overcloud-compute-0 heat-admin]# journalctl -fn 100000000 | grep "FOOO"
Mar 24 11:51:06 overcloud-compute-0.localdomain os-collect-config[5168]: [2016-03-24 11:51:06,583] (heat-config) [INFO] {"deploy_stdout": "FOOOO from overcloud-compute-0.localdomain\n", "deploy_stderr": "", "deploy_status_code": 0}
Can you please include the full deploy commands you used?
--- Additional comment from Adriano Petrich on 2016-03-24 12:19:14 EDT ---
So the overcloud was deployed with
source /home/stack/stackrc; openstack overcloud deploy --debug --log-file overcloud_deployment_16.log --templates /home/stack/local_tht/ --libvirt-type=qemu --ntp-server 10.5.26.10 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout=90 -e /home/stack/local_tht/environments/storage-environment.yaml -e ~/default-overcloud-settings.yaml
$ cat default-overcloud-settings.yaml
parameters:
CinderLVMLoopDeviceSize: 10000
trying to run a foo.yml template that does only pipe echo foo into a file with this command
openstack overcloud deploy --debug --templates /home/stack/local_tht/ --libvirt-type=qemu -e /home/stack/local_tht/environments/storage-environment.yaml -e ~/default-overcloud-settings.yaml -e foo.yml
Still fails
the debug message is pretty generic
Stack overcloud UPDATE_FAILED
Heat Stack update failed.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
self.take_action(parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 886, in take_action
self._deploy_tripleo_heat_templates(stack, parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 355, in _deploy_tripleo_heat_templates
environments, parsed_args.timeout)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 281, in _heat_deploy
raise Exception("Heat Stack update failed.")
Exception: Heat Stack update failed.
clean_up DeployOvercloud: Heat Stack update failed.
cat /var/log/heat/heat-engine.log | grep ERROR
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource yield
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource prop_diff])
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource step = next(subtask)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource handler_data = handler(*args)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 427, in handle_update
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource checkers[0].start()
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource self.step()
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource next(self._runner)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource timeout)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource self.raise_local_exception(ex)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource raise exception.ResourceFailure(message, self, action=self.action)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource ResourceFailure: resources.Compute: "u'1:1000'" is not a list
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource
heat stack-show overcloud gives
| stack_name | overcloud
| stack_owner | admin
| stack_status | UPDATE_FAILED
| stack_status_reason | resources.Compute: "u'1:1000'" is not a list
I'm having the same error if I try to do an osp7 install and then upgrade only the undercloud or if I do a mixed deployment (a fresh undercloud osp8 deploy and deploy osp7 overcloud images on it)
--- Additional comment from Zane Bitter on 2016-03-29 10:29:27 EDT ---
Can you attach the full log?
--- Additional comment from Adriano Petrich on 2016-03-29 11:34 EDT ---
Sure. here is the heat log for a mixed deployment
--- Additional comment from Adriano Petrich on 2016-03-29 11:37:00 EDT ---
here is output of heat event-list overcloud
18:52:42 +------------------------+--------------------------------------+----------------------------------------------------------------------+--------------------+---------------------+
18:52:42 | resource_name | id | resource_status_reason | resource_status | event_time |
18:52:42 +------------------------+--------------------------------------+----------------------------------------------------------------------+--------------------+---------------------+
18:52:42 | overcloud | 6807a432-2836-493d-b5a4-62dc7540a98b | Stack CREATE started | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | MysqlClusterUniquePart | 566ae3eb-e0d2-47ab-ba98-0c2445e51655 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | RabbitCookie | bae96c91-1ba0-4f5c-992e-3491debaf37a | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | Networks | 87d94cf5-c7e4-4b3b-9c5e-234eaf101093 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | PcsdPassword | 70a537db-7b3a-4991-8b25-2156e3e95750 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | VipConfig | 5e6959ce-4e08-4050-b08d-9870c11cd5e9 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | HorizonSecret | e8ba57ef-d2f8-4ce1-99c1-bc8395d25f38 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | HeatAuthEncryptionKey | f6860df8-34ee-44ab-bf7d-5cf6bce8c015 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | MysqlRootPassword | 404c6b25-0834-4ace-8ad1-071f811a197b | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | MysqlClusterUniquePart | 4051abd1-ea88-41a5-bd81-2845b6217718 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | RabbitCookie | 9bb3d994-ce57-46f3-9c96-521da2fe7f4f | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | PcsdPassword | f0f85330-f172-4342-b747-5710efbf709c | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | VipConfig | 329ee3d8-7694-465f-9d15-c9442b8db550 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | HorizonSecret | d43a7bc7-dcb1-4600-aa7d-842fafa2d76c | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | HeatAuthEncryptionKey | 3cf8b533-5224-4250-9211-82146dddba2e | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | MysqlRootPassword | b2814b6b-1dec-46b3-b9a2-b27472f5460a | state changed | CREATE_COMPLETE | 2016-03-23T18:47:18 |
18:52:42 | Networks | 43d8a720-194f-45b8-8ceb-531d36e87194 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:22 |
18:52:42 | ControlVirtualIP | bf06caff-feea-4a54-8bb2-f633213b511c | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:22 |
18:52:42 | ObjectStorage | 2f0cbf19-8d81-41a3-b51c-e7908b75f19d | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:22 |
18:52:42 | CephStorage | 7207d0fc-b7bd-4777-9162-4df0a610dff8 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:23 |
18:52:42 | ControlVirtualIP | 82f2ed09-da58-4666-b836-3420e99e934f | state changed | CREATE_COMPLETE | 2016-03-23T18:47:26 |
18:52:42 | ObjectStorage | a9186cdc-10a6-42c7-a127-26c1ee87cd52 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:26 |
18:52:42 | PublicVirtualIP | 91feef9d-c7da-40c4-b4db-253217d8f17d | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:27 |
18:52:42 | StorageMgmtVirtualIP | 6d0a171c-6007-441b-83d9-a37b168be050 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:28 |
18:52:42 | RedisVirtualIP | a2912740-3cf1-45d7-9463-c551c2e856c5 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:28 |
18:52:42 | StorageVirtualIP | 7032ccdc-8d4f-4ea8-9fa2-3d667f30eeba | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:29 |
18:52:42 | InternalApiVirtualIP | 040383c5-20c5-494d-bca9-a7327f7f677f | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:30 |
18:52:42 | PublicVirtualIP | d590dd4c-07e9-4a20-bfd5-ca36ed63af12 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:32 |
18:52:42 | StorageMgmtVirtualIP | f06b5e2c-7527-41f1-9ca6-fef464922510 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:32 |
18:52:42 | RedisVirtualIP | 1115bfe6-dcd6-4326-b691-d1f40587856e | state changed | CREATE_COMPLETE | 2016-03-23T18:47:33 |
18:52:42 | StorageVirtualIP | 828f7b8d-5866-4e0b-8b08-91c0b16d1ffc | state changed | CREATE_COMPLETE | 2016-03-23T18:47:33 |
18:52:42 | InternalApiVirtualIP | 9d1e8535-48c7-4882-a0e6-008194345a77 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:33 |
18:52:42 | VipMap | ef86ce52-87c8-4462-9336-ff95c9ffd271 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:33 |
18:52:42 | VipMap | e0d049cd-cc91-4c16-8050-58bfa4f2ba81 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:35 |
18:52:42 | EndpointMap | cd6ea216-009a-4878-96c1-91aa9422bab3 | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:35 |
18:52:42 | EndpointMap | 852e74fe-f5ae-47cb-9933-045fc2a03086 | state changed | CREATE_COMPLETE | 2016-03-23T18:47:37 |
18:52:42 | Compute | 25590ec9-a91d-4f2a-a3cc-92d5b53e12cd | state changed | CREATE_IN_PROGRESS | 2016-03-23T18:47:37 |
18:52:42 | Compute | b77b9b8f-adc4-42f6-974b-e7d813ed8572 | resources.Compute: "u'1:1000'" is not a list | CREATE_FAILED | 2016-03-23T18:47:38 |
18:52:42 | CephStorage | 7b8c911f-1b40-4210-bc7d-3e9a7afee7fa | CREATE aborted | CREATE_FAILED | 2016-03-23T18:51:39 |
18:52:42 | overcloud | ba6f1a65-302c-4b60-9ccc-aab3d215567c | Resource CREATE failed: resources.Compute: "u'1:1000'" is not a list | CREATE_FAILED | 2016-03-23T18:51:39 |
18:52:42 +------------------------+--------------------------------------+----------------------------------------------------------------------+--------------------+---------------------+
18:52:42
--- Additional comment from Zane Bitter on 2016-03-29 19:05:02 EDT ---
The interesting backtrace is the first one:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 674, in validate
result = res.validate()
File "/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py", line 279, in validate
return super(TemplateResource, self).validate()
File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 57, in validate
super(StackResource, self).validate()
File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1135, in validate
return self.validate_template()
File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1152, in validate_template
with_value=self.stack.strict_validate)
File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 383, in validate
self._get_property_value(key, validate=True)
File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 451, in _get_property_value
return prop.get_value(None, validate)
File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 326, in get_value
_value = self._get_list(value, validate)
File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 296, in _get_list
raise TypeError(_('"%s" is not a list') % repr(value))
TypeError: "u'1:1000'" is not a list
It appears to be a problem with validating the properties that are inferred from a nested template. The properties validation code is in general designed to fail hard, because normally the properties schema is hard-coded in the resource plugin in Heat. However, in the case of a template resource we generate the schema on the fly based on the input parameters of the template.
When validating the properties of the template resource, any values explicitly passed will generally be lists - e.g. get_param on a comma_delimited_list parameter will produce a list to pass.
The problem in this case is the *default* value. This is obtained directly from the template and not subject to conversion (i.e. splitting on commas) even though it would be when it is used as a parameter.
A workaround would be to always specify the default value as a JSON list. It appears we are already doing that in the 8.0 t-h-t templates, although we didn't in the 7.x templates. So I'm not sure why you're seeing this after upgrading to OSP8.
You're correct that this is a different issue from bug 1303084, which was hitting a similar validation problem on the same data when trying to validate the old property values against the new parameter schema (which only happens after a failure of some sort leaves the stack in a kind of intermediate state).
--- Additional comment from Mike Burns on 2016-03-29 19:09:26 EDT ---
(In reply to Zane Bitter from comment #9)
> A workaround would be to always specify the default value as a JSON list. It
> appears we are already doing that in the 8.0 t-h-t templates, although we
> didn't in the 7.x templates. So I'm not sure why you're seeing this after
> upgrading to OSP8.
This case is managing a 7.x overcloud with 8.0 undercloud. The 7.x overcloud is still using the 7.x templates.
--- Additional comment from Adriano Petrich on 2016-03-30 06:29:50 EDT ---
Looking at this patch https://review.openstack.org/gitweb?p=openstack/tripleo-heat-templates.git;a=commit;h=be0d3f3520037c4e2e9fb4f9a235171aaad7e253
I found that I could not apply that to the 7 version of tht because most of the stuff from the patch is already there
but there's a difference
on the patch the defaults are marked as ["1:1000",] and on tht7 is "1:1000"
so I did a patch to add the extra square brackets and that error vanished and 10 minutes into deploy we got a new one
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource yield
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource prop_diff])
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 309, in wrapper
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource step = next(subtask)
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 682, in action_handler_task
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource while not check(handler_data):
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 456, in check_update_complete
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource return self._check_complete()
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 294, in _check_complete
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource raise exc
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource yield
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource prop_diff])
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 309, in wrapper
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource step = next(subtask)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 682, in action_handler_task
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource while not check(handler_data):
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 400, in check_update_complete
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource if not checker.step():
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource next(self._runner)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 388, in _run_to_completion
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource self).check_update_complete(updater):
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 442, in check_update_complete
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource cookie=cookie)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 372, in _check_status_complete
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource action=action)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource ResourceFailure: resources.ControllerOvercloudServicesDeployment_Step4: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource yield
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource prop_diff])
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 309, in wrapper
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource step = next(subtask)
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 682, in action_handler_task
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource while not check(handler_data):
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 442, in check_update_complete
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource cookie=cookie)
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 372, in _check_status_complete
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource action=action)
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource ResourceFailure: resources.ControllerNodesPostDeployment: resources.ControllerOvercloudServicesDeployment_Step4: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource
--- Additional comment from Adriano Petrich on 2016-03-30 06:31 EDT ---
--- Additional comment from Giulio Fidente on 2016-03-30 06:45:52 EDT ---
(In reply to Zane Bitter from comment #9)
> A workaround would be to always specify the default value as a JSON list. It
> appears we are already doing that in the 8.0 t-h-t templates, although we
> didn't in the 7.x templates. So I'm not sure why you're seeing this after
> upgrading to OSP8.
Given comment #10 we can update the kilo templates to use the brackets but it won't be effective for those not using stock templates.
Zane, could we, in addition to updating the kilo templates and document usage of an environment file as a workaround?
parameter_defaults:
NeutronTunnelIdRanges: ['1:1000', ]
NeutronVniRanges: ['1:1000', ]
Or do you think a 'backward compatibility' update for Heat would be doable?
--- Additional comment from Zane Bitter on 2016-03-30 11:36:28 EDT ---
Ah, I read this several times and never grokked that only the undercloud had been updated. That explains it :)
I wasn't aware that we added more rigorous validation in Liberty, but apparently that must be the case. The workaround (of using ["1:1000"] as the default) will certainly solve it for now, but I'll also clone this bug to the openstack-heat component and implement a proper fix in Heat itself.
The 'error' in comment #11 is a bog-standard Puppet failure, completely unrelated.