Bug 1320454 - Any changes on a mixed version deploy fails with "u'1:1000'" is not a list
Summary: Any changes on a mixed version deploy fails with "u'1:1000'" is not a list
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 8.0 (Liberty)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ga
: 8.0 (Liberty)
Assignee: Marios Andreou
QA Contact: Dan Yasny
URL:
Whiteboard:
Depends On:
Blocks: 1322619
TreeView+ depends on / blocked
 
Reported: 2016-03-23 09:30 UTC by Adriano Petrich
Modified: 2016-04-07 21:49 UTC (History)
12 users (show)

Fixed In Version: openstack-tripleo-heat-templates-0.8.14-2.el7ost
Doc Type: Bug Fix
Doc Text:
Stricter validation in Red Hat OpenStack Platform 8's Orchestration service (heat) caused the Overcloud stack update to fail from an upgraded Undercloud with the following error: ERROR heat.engine.resource ResourceFailure: resources.Compute: "u'1:1000'" is not a list. This fix which properly formats the NeutronVniRanges parameter to include the required '[]' has been backported to the OpenStack Platform 7 openstack-tripleo-heat-templates package and should be available as of openstack-tripleo-heat-templates-kilo-0.8.14-5.el7ost.noarch. Now stack updates of the Overcloud stack will not fail with the error when using an upgraded Undercloud to manage an existing Overcloud while using the version 7 templates (which are located at /usr/share/openstack-tripleo-heat-templates/kilo).
Clone Of:
: 1322619 (view as bug list)
Environment:
Last Closed: 2016-04-07 21:49:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Heat engine log (1.28 MB, text/plain)
2016-03-29 15:34 UTC, Adriano Petrich 		no flags Details
Add square brackets (4.39 KB, patch)
2016-03-30 10:31 UTC, Adriano Petrich 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0604 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 8 director Enhancement Advisory 2016-04-08 01:03:56 UTC

Description Adriano Petrich 2016-03-23 09:30:21 UTC 
Description of problem:
This is a mixed version problem. It happens when you do a full osp7 install (undercloud and overcloud) then you just upgrade the undercloud to osp8.  After you do that things work just fine, you can create and delete instances on the overcloud and so on, but if you try to run any scale or just a simple template via tht it fails and everything gets borked on a UPDATE_FAIL status



Version-Release number of selected component (if applicable):


How reproducible:
Very

Steps to Reproduce:
1. do a full osp7 install
2. upgrade the undercloud as per the kbase document 
3. try run any template  openstack overcloud deploy --templates ~/local_templates -e foo.yaml

[stack@undercloud ~]$cat foo.yaml 
resource_registry:
  OS::TripleO::NodeExtraConfig: install_nmap_template.yaml

  
[stack@undercloud ~]$ cat install_nmap_template.yaml 
heat_template_version: 2014-10-16
description: 'Install Nmap'
parameters:
  server:
    type: string

resources:

  NmapConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config: |
        #!/bin/sh
        echo "nameserver 8.8.8.8" >> /etc/resolv.conf
        yum -y install nmap

  NmapDeployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      name: NmapDeployment
      config: {get_resource: NmapConfig}
      server: {get_param: server}

outputs:
  deploy_stdout:
    value: "None"




Actual results:
It fails

Expected results:
it should pass

Additional info:

2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack [req-a85dddaa-68eb-47b0-89f8-92e066e6e4d4 9b9513cbe51c442fb8bac58d251133cc d73003a5c802451693a4e2e12ab91323] Exception: "u'1:1000'" is not a list
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack Traceback (most recent call last):
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 674, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     result = res.validate()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py", line 279, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     return super(TemplateResource, self).validate()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 57, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     super(StackResource, self).validate()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1135, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     return self.validate_template()
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1152, in validate_template
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     with_value=self.stack.strict_validate)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 383, in validate
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     self._get_property_value(key, validate=True)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 451, in _get_property_value
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     return prop.get_value(None, validate)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 326, in get_value
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     _value = self._get_list(value, validate)
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 296, in _get_list
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack     raise TypeError(_('"%s" is not a list') % repr(value))
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack TypeError: "u'1:1000'" is not a list
2016-03-22 07:04:36.212 17245 ERROR heat.engine.stack 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource [-] update_stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource Traceback (most recent call last):
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 434, in update_with_template
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     args)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 269, in update_stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     args=args))
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 61, in call
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     return client.call(ctxt, method, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 403, in call
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     return self.prepare().call(ctxt, method, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     retry=self.retry)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     timeout=timeout, retry=retry)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     retry=retry)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     raise result
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource StackValidationFailed_Remote: "u'1:1000'" is not a list
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource Traceback (most recent call last):
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 308, in wrapped
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     return func(self, ctx, *args, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 868, in update_stack
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     cnxt, current_stack, template, params, files, args)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 830, in _prepare_stack_updates
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     updated_stack.validate()
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     return f(*args, **kwargs)
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 685, in validate
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource     message=encodeutils.safe_decode(six.text_type(ex)))
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource StackValidationFailed: "u'1:1000'" is not a list
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.231 17244 ERROR heat.engine.resources.stack_resource 
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     yield
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     prop_diff])
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     step = next(subtask)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     handler_data = handler(*args)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 427, in handle_update
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     checkers[0].start()
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     self.step()
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     next(self._runner)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     timeout)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     self.raise_local_exception(ex)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource     raise exception.ResourceFailure(message, self, action=self.action)
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource ResourceFailure: resources.Compute: "u'1:1000'" is not a list
2016-03-22 07:04:36.236 17244 ERROR heat.engine.resource
Comment 2 Adriano Petrich 2016-03-23 10:49:13 UTC 
Just for completeness 

this bug looks a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1303084

but it is not the same.

I verified and the fixes for that bug were applied on my env such as this one: 
https://review.openstack.org/#/c/286874/1/heat/engine/resource.py
Comment 3 Dan Yasny 2016-03-23 14:07:22 UTC 
Reproduced again in manual testing
Comment 4 Marios Andreou 2016-03-24 12:54:55 UTC 
wasn't able to reproduce this; how long before this failure happens (ish, I mean is it immediate?). Here is what I did ftr (in particular, are you sure you included all the environment files for the stack update?):


deploy HA net-iso overcloud like: 

openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server "0.fedora.pool.ntp.org"

update packages on the undercloud and rerun openstack undercloud install.

I used your example templates, (but just echo "FOOOO from `hostname`" rather than install nmap):

openstack overcloud deploy --templates /usr/share/openstack-tripleo-heat-templates -e  /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e foo.yaml

This ran (almost) to completion, I hit an error but I think it is because I restarted openstack-nova-api mid-way through the update (forgot to after the undercloud update). In particular I didn't see the 
"u'1:1000'" is not a list and on the compute/controller I see my 'config' has run OK



[root@overcloud-controller-0 ~]# journalctl -fn 1000000 | grep "FOOOO"
Mar 24 11:51:57 overcloud-controller-0.localdomain os-collect-config[5053]: [2016-03-24 11:51:57,215] (heat-config) [INFO] {"deploy_stdout": "FOOOO from overcloud-controller-0.localdomain\n", "deploy_stderr": "", "deploy_status_code": 0}


[root@overcloud-compute-0 heat-admin]# journalctl -fn 100000000 | grep "FOOO"
Mar 24 11:51:06 overcloud-compute-0.localdomain os-collect-config[5168]: [2016-03-24 11:51:06,583] (heat-config) [INFO] {"deploy_stdout": "FOOOO from overcloud-compute-0.localdomain\n", "deploy_stderr": "", "deploy_status_code": 0}


Can you please include the full deploy commands you used?
Comment 5 Adriano Petrich 2016-03-24 16:19:14 UTC 
So the overcloud was deployed with

source /home/stack/stackrc; openstack overcloud deploy --debug --log-file overcloud_deployment_16.log --templates /home/stack/local_tht/ --libvirt-type=qemu --ntp-server 10.5.26.10 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout=90 -e /home/stack/local_tht/environments/storage-environment.yaml -e ~/default-overcloud-settings.yaml

$ cat default-overcloud-settings.yaml 
parameters:
  CinderLVMLoopDeviceSize: 10000


trying to run a foo.yml template that does only pipe echo foo into a file with this command

openstack overcloud deploy --debug --templates /home/stack/local_tht/ --libvirt-type=qemu -e /home/stack/local_tht/environments/storage-environment.yaml -e ~/default-overcloud-settings.yaml -e foo.yml


Still fails

the debug message is pretty generic

Stack overcloud UPDATE_FAILED
Heat Stack update failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 886, in take_action
    self._deploy_tripleo_heat_templates(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 355, in _deploy_tripleo_heat_templates
    environments, parsed_args.timeout)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 281, in _heat_deploy
    raise Exception("Heat Stack update failed.")
Exception: Heat Stack update failed.
clean_up DeployOvercloud: Heat Stack update failed.


cat /var/log/heat/heat-engine.log | grep ERROR 

2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     yield
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     prop_diff])
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     step = next(subtask)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     handler_data = handler(*args)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 427, in handle_update
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     checkers[0].start()
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     self.step()
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     next(self._runner)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     timeout)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     self.raise_local_exception(ex)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource     raise exception.ResourceFailure(message, self, action=self.action)
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource ResourceFailure: resources.Compute: "u'1:1000'" is not a list
2016-03-24 12:09:48.494 31039 ERROR heat.engine.resource 

heat stack-show overcloud gives

| stack_name            | overcloud
| stack_owner           | admin
| stack_status          | UPDATE_FAILED
| stack_status_reason   | resources.Compute: "u'1:1000'" is not a list


I'm having the same error if I try to do an osp7 install and then upgrade only the undercloud or if I do a mixed deployment (a fresh undercloud osp8 deploy and deploy osp7 overcloud images on it)
Comment 6 Zane Bitter 2016-03-29 14:29:27 UTC 
Can you attach the full log?
Comment 7 Adriano Petrich 2016-03-29 15:34:56 UTC 
Created attachment 1141324 [details]
Heat engine log

Sure. here is the heat log for a mixed deployment
Comment 8 Adriano Petrich 2016-03-29 15:37:00 UTC 
here is output of heat event-list overcloud

18:52:42 +------------------------+--------------------------------------+----------------------------------------------------------------------+--------------------+---------------------+
18:52:42 | resource_name          | id                                   | resource_status_reason                                               | resource_status    | event_time          |
18:52:42 +------------------------+--------------------------------------+----------------------------------------------------------------------+--------------------+---------------------+
18:52:42 | overcloud              | 6807a432-2836-493d-b5a4-62dc7540a98b | Stack CREATE started                                                 | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | MysqlClusterUniquePart | 566ae3eb-e0d2-47ab-ba98-0c2445e51655 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | RabbitCookie           | bae96c91-1ba0-4f5c-992e-3491debaf37a | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | Networks               | 87d94cf5-c7e4-4b3b-9c5e-234eaf101093 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:16 |
18:52:42 | PcsdPassword           | 70a537db-7b3a-4991-8b25-2156e3e95750 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | VipConfig              | 5e6959ce-4e08-4050-b08d-9870c11cd5e9 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | HorizonSecret          | e8ba57ef-d2f8-4ce1-99c1-bc8395d25f38 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | HeatAuthEncryptionKey  | f6860df8-34ee-44ab-bf7d-5cf6bce8c015 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | MysqlRootPassword      | 404c6b25-0834-4ace-8ad1-071f811a197b | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:17 |
18:52:42 | MysqlClusterUniquePart | 4051abd1-ea88-41a5-bd81-2845b6217718 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | RabbitCookie           | 9bb3d994-ce57-46f3-9c96-521da2fe7f4f | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | PcsdPassword           | f0f85330-f172-4342-b747-5710efbf709c | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | VipConfig              | 329ee3d8-7694-465f-9d15-c9442b8db550 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | HorizonSecret          | d43a7bc7-dcb1-4600-aa7d-842fafa2d76c | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | HeatAuthEncryptionKey  | 3cf8b533-5224-4250-9211-82146dddba2e | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | MysqlRootPassword      | b2814b6b-1dec-46b3-b9a2-b27472f5460a | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:18 |
18:52:42 | Networks               | 43d8a720-194f-45b8-8ceb-531d36e87194 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:22 |
18:52:42 | ControlVirtualIP       | bf06caff-feea-4a54-8bb2-f633213b511c | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:22 |
18:52:42 | ObjectStorage          | 2f0cbf19-8d81-41a3-b51c-e7908b75f19d | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:22 |
18:52:42 | CephStorage            | 7207d0fc-b7bd-4777-9162-4df0a610dff8 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:23 |
18:52:42 | ControlVirtualIP       | 82f2ed09-da58-4666-b836-3420e99e934f | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:26 |
18:52:42 | ObjectStorage          | a9186cdc-10a6-42c7-a127-26c1ee87cd52 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:26 |
18:52:42 | PublicVirtualIP        | 91feef9d-c7da-40c4-b4db-253217d8f17d | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:27 |
18:52:42 | StorageMgmtVirtualIP   | 6d0a171c-6007-441b-83d9-a37b168be050 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:28 |
18:52:42 | RedisVirtualIP         | a2912740-3cf1-45d7-9463-c551c2e856c5 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:28 |
18:52:42 | StorageVirtualIP       | 7032ccdc-8d4f-4ea8-9fa2-3d667f30eeba | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:29 |
18:52:42 | InternalApiVirtualIP   | 040383c5-20c5-494d-bca9-a7327f7f677f | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:30 |
18:52:42 | PublicVirtualIP        | d590dd4c-07e9-4a20-bfd5-ca36ed63af12 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:32 |
18:52:42 | StorageMgmtVirtualIP   | f06b5e2c-7527-41f1-9ca6-fef464922510 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:32 |
18:52:42 | RedisVirtualIP         | 1115bfe6-dcd6-4326-b691-d1f40587856e | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:33 |
18:52:42 | StorageVirtualIP       | 828f7b8d-5866-4e0b-8b08-91c0b16d1ffc | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:33 |
18:52:42 | InternalApiVirtualIP   | 9d1e8535-48c7-4882-a0e6-008194345a77 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:33 |
18:52:42 | VipMap                 | ef86ce52-87c8-4462-9336-ff95c9ffd271 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:33 |
18:52:42 | VipMap                 | e0d049cd-cc91-4c16-8050-58bfa4f2ba81 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:35 |
18:52:42 | EndpointMap            | cd6ea216-009a-4878-96c1-91aa9422bab3 | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:35 |
18:52:42 | EndpointMap            | 852e74fe-f5ae-47cb-9933-045fc2a03086 | state changed                                                        | CREATE_COMPLETE    | 2016-03-23T18:47:37 |
18:52:42 | Compute                | 25590ec9-a91d-4f2a-a3cc-92d5b53e12cd | state changed                                                        | CREATE_IN_PROGRESS | 2016-03-23T18:47:37 |
18:52:42 | Compute                | b77b9b8f-adc4-42f6-974b-e7d813ed8572 | resources.Compute: "u'1:1000'" is not a list                         | CREATE_FAILED      | 2016-03-23T18:47:38 |
18:52:42 | CephStorage            | 7b8c911f-1b40-4210-bc7d-3e9a7afee7fa | CREATE aborted                                                       | CREATE_FAILED      | 2016-03-23T18:51:39 |
18:52:42 | overcloud              | ba6f1a65-302c-4b60-9ccc-aab3d215567c | Resource CREATE failed: resources.Compute: "u'1:1000'" is not a list | CREATE_FAILED      | 2016-03-23T18:51:39 |
18:52:42 +------------------------+--------------------------------------+----------------------------------------------------------------------+--------------------+---------------------+
18:52:42
Comment 9 Zane Bitter 2016-03-29 23:05:02 UTC 
The interesting backtrace is the first one:

 Traceback (most recent call last):
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 674, in validate
     result = res.validate()
   File "/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py", line 279, in validate
     return super(TemplateResource, self).validate()
   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 57, in validate
     super(StackResource, self).validate()
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1135, in validate
     return self.validate_template()
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1152, in validate_template
     with_value=self.stack.strict_validate)
   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 383, in validate
     self._get_property_value(key, validate=True)
   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 451, in _get_property_value
     return prop.get_value(None, validate)
   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 326, in get_value
     _value = self._get_list(value, validate)
   File "/usr/lib/python2.7/site-packages/heat/engine/properties.py", line 296, in _get_list
     raise TypeError(_('"%s" is not a list') % repr(value))
 TypeError: "u'1:1000'" is not a list


It appears to be a problem with validating the properties that are inferred from a nested template. The properties validation code is in general designed to fail hard, because normally the properties schema is hard-coded in the resource plugin in Heat. However, in the case of a template resource we generate the schema on the fly based on the input parameters of the template.

When validating the properties of the template resource, any values explicitly passed will generally be lists - e.g. get_param on a comma_delimited_list parameter will produce a list to pass.

The problem in this case is the *default* value. This is obtained directly from the template and not subject to conversion (i.e. splitting on commas) even though it would be when it is used as a parameter.

A workaround would be to always specify the default value as a JSON list. It appears we are already doing that in the 8.0 t-h-t templates, although we didn't in the 7.x templates. So I'm not sure why you're seeing this after upgrading to OSP8.

You're correct that this is a different issue from bug 1303084, which was hitting a similar validation problem on the same data when trying to validate the old property values against the new parameter schema (which only happens after a failure of some sort leaves the stack in a kind of intermediate state).
Comment 10 Mike Burns 2016-03-29 23:09:26 UTC 
(In reply to Zane Bitter from comment #9)

> A workaround would be to always specify the default value as a JSON list. It
> appears we are already doing that in the 8.0 t-h-t templates, although we
> didn't in the 7.x templates. So I'm not sure why you're seeing this after
> upgrading to OSP8.

This case is managing a 7.x overcloud with 8.0 undercloud.  The 7.x overcloud is still using the 7.x templates.
Comment 11 Adriano Petrich 2016-03-30 10:29:50 UTC 
Looking at this patch https://review.openstack.org/gitweb?p=openstack/tripleo-heat-templates.git;a=commit;h=be0d3f3520037c4e2e9fb4f9a235171aaad7e253 

I found that I could not apply that to the 7 version of tht because most of the stuff from the patch is already there

but there's a difference

on the patch the defaults are marked as ["1:1000",] and on tht7 is "1:1000"

so I did a patch to add the extra square brackets and that error vanished and 10 minutes into deploy we got a new one 



2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource     yield
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource     prop_diff])
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 309, in wrapper
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource     step = next(subtask)
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 682, in action_handler_task
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource     while not check(handler_data):
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 456, in check_update_complete
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource     return self._check_complete()
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 294, in _check_complete
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource     raise exc
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2016-03-30 06:09:04.647 17244 ERROR heat.engine.resource 
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     yield
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     prop_diff])
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 309, in wrapper
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     step = next(subtask)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 682, in action_handler_task
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     while not check(handler_data):
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 400, in check_update_complete
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     if not checker.step():
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     next(self._runner)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 388, in _run_to_completion
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     self).check_update_complete(updater):
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 442, in check_update_complete
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     cookie=cookie)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 372, in _check_status_complete
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource     action=action)
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource ResourceFailure: resources.ControllerOvercloudServicesDeployment_Step4: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2016-03-30 06:09:06.697 17245 ERROR heat.engine.resource 
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource Traceback (most recent call last):
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource     yield
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 968, in update
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource     prop_diff])
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 309, in wrapper
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource     step = next(subtask)
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 682, in action_handler_task
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource     while not check(handler_data):
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 442, in check_update_complete
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource     cookie=cookie)
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 372, in _check_status_complete
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource     action=action)
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource ResourceFailure: resources.ControllerNodesPostDeployment: resources.ControllerOvercloudServicesDeployment_Step4: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2016-03-30 06:09:08.631 17244 ERROR heat.engine.resource
Comment 12 Adriano Petrich 2016-03-30 10:31:03 UTC 
Created attachment 1141675 [details]
Add square brackets
Comment 13 Giulio Fidente 2016-03-30 10:45:52 UTC 
(In reply to Zane Bitter from comment #9)
> A workaround would be to always specify the default value as a JSON list. It
> appears we are already doing that in the 8.0 t-h-t templates, although we
> didn't in the 7.x templates. So I'm not sure why you're seeing this after
> upgrading to OSP8.

Given comment #10 we can update the kilo templates to use the brackets but it won't be effective for those not using stock templates.

Zane, could we, in addition to updating the kilo templates and document usage of an environment file as a workaround?

parameter_defaults:
  NeutronTunnelIdRanges: ['1:1000', ]
  NeutronVniRanges: ['1:1000', ]

Or do you think a 'backward compatibility' update for Heat would be doable?
Comment 14 Zane Bitter 2016-03-30 15:36:28 UTC 
Ah, I read this several times and never grokked that only the undercloud had been updated. That explains it :)

I wasn't aware that we added more rigorous validation in Liberty, but apparently that must be the case. The workaround (of using ["1:1000"] as the default) will certainly solve it for now, but I'll also clone this bug to the openstack-heat component and implement a proper fix in Heat itself.

The 'error' in comment #11 is a bog-standard Puppet failure, completely unrelated.
Comment 16 Dan Yasny 2016-04-04 14:40:56 UTC 
Tested on current puddle versions:
openstack-tripleo-heat-templates-kilo-0.8.14-5.el7ost.noarch
openstack-heat-templates-0-0.1.20151019.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-5.el7ost.noarch
Comment 18 errata-xmlrpc 2016-04-07 21:49:51 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html
Note You need to log in before you can comment on or make changes to this bug.