Bug 1310879

+++ This bug was initially created as a clone of Bug #1300847 +++

rhel-osp-director: Overcloud update from 7.2-> 7.3 fails: "resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned"


Environment:
openstack-swift-container-2.3.0-2.el7ost.noarch
python-swiftclient-2.4.0-1.el7ost.noarch
openstack-swift-object-2.3.0-2.el7ost.noarch
openstack-swift-proxy-2.3.0-2.el7ost.noarch
instack-undercloud-2.1.2-37.el7ost.noarch
openstack-swift-account-2.3.0-2.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-106.el7ost.noarch
openstack-swift-2.3.0-2.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud 7.1 with director.
2. Register the nodes with Sat5 (pointing to 7.3).
3. Update the undercloud
4. Attempt  to update the OC nodes.


Result:

After running some time and actually yum updating the nodes, the deployment fails:
...
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED


Doing some debuging points to SwiftDevicesAndProxyConfig:

[stack@undercloud ~]$ heat resource-list overcloud|grep -v COMPLE
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time         |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| SwiftDevicesAndProxyConfig                | 515b477a-4261-4b10-bb53-4d9adae478eb          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | UPDATE_FAILED   | 2016-01-21T21:53:39Z |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+


Additional debugging shows: resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned



[stack@undercloud ~]$ heat resource-show overcloud SwiftDevicesAndProxyConfig
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                              |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                  |
|                        |   "config_id": "1f069129-cd23-4761-ac8a-769847c8163b"                                                                                                              |
|                        | }                                                                                                                                                                  |
| description            |                                                                                                                                                                    |
| links                  | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud/9799dd92-c319-4878-a605-bda9dd5ea6d0/resources/SwiftDevicesAndProxyConfig (self)      |
|                        | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud/9799dd92-c319-4878-a605-bda9dd5ea6d0 (stack)                                          |
|                        | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud-SwiftDevicesAndProxyConfig-dbbgtzmy7ewg/515b477a-4261-4b10-bb53-4d9adae478eb (nested) |
| logical_resource_id    | SwiftDevicesAndProxyConfig                                                                                                                                         |
| physical_resource_id   | 515b477a-4261-4b10-bb53-4d9adae478eb                                                                                                                               |
| required_by            | ObjectStorageSwiftDeployment                                                                                                                                       |
|                        | ControllerSwiftDeployment                                                                                                                                          |
| resource_name          | SwiftDevicesAndProxyConfig                                                                                                                                         |
| resource_status        | UPDATE_FAILED                                                                                                                                                      |
| resource_status_reason | ValueError: resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned                                                        |
| resource_type          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig                                                                                                                  |
| updated_time           | 2016-01-21T21:53:39Z                                                                                                                                               |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+


Expected result:
completed update.

--- Additional comment from James Slagle on 2016-01-22 07:49:47 EST ---

In 7.2, the resource looked like:

  SwiftDevicesAndProxyConfig:
    type: OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig
    properties:
      controller_swift_devices: {get_attr: [Controller, swift_device]}
      object_store_swift_devices: {get_attr: [ObjectStorage, swift_device]}
      controller_swift_proxy_memcaches: {get_attr: [Controller, swift_proxy_memcache]}

In 7.3, we've added some *_v6 properties (which should always be set):

  SwiftDevicesAndProxyConfig:
    type: OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig
    properties:
      controller_swift_devices: {get_attr: [Controller, swift_device]}
      controller_swift_devices_v6: {get_attr: [Controller, swift_device_v6]}
      object_store_swift_devices: {get_attr: [ObjectStorage, swift_device]}
      object_store_swift_devices_v6: {get_attr: [ObjectStorage, swift_device_v6]}
      controller_swift_proxy_memcaches: {get_attr: [Controller, swift_proxy_memcache]}
      controller_swift_proxy_memcaches_v6: {get_attr: [Controller, swift_proxy_memcache_v6]}

Also in 7.3 the *_v6 attributes are *new* outputs on the Controller resource. So I'm wondering if we're hitting some slight nuance here with the Update logic since this is failing during an update. Do we need to add an explicit depends_on: Controller on the SwiftDevicesAndProxyConfig resource? Would that ensure that Controller is updated first and provides the new outputs (assuming that's even what's triggering the error message)?

--- Additional comment from Zane Bitter on 2016-02-09 13:39:51 EST ---

Thomas has established that this occurs when:

* You update the stack with a child template that adds new mandatory properties
* The update fails for any reason _before_ the point of updating this resource
* You run another update