Bug 1310879

Summary: rhel-osp-director: Overcloud update from 7.2-> 7.3 fails "resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned "
Product: Red Hat OpenStack Reporter: Zane Bitter <zbitter>
Component: openstack-heatAssignee: Zane Bitter <zbitter>
Status: CLOSED CURRENTRELEASE QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 8.0 (Liberty)CC: adahms, athomas, jcoufal, jschluet, jslagle, mburns, mcornea, ohochman, rhel-osp-director-maint, rybrown, sasha, sbaker, shardy, srevivo, therve, zbitter
Target Milestone: gaKeywords: TestOnly, ZStream
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-heat-5.0.1-2.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1300847 Environment:
Last Closed: 2016-04-28 13:51:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1300847    
Bug Blocks:    

Description Zane Bitter 2016-02-22 21:19:46 UTC
+++ This bug was initially created as a clone of Bug #1300847 +++

rhel-osp-director: Overcloud update from 7.2-> 7.3 fails: "resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned"


Environment:
openstack-swift-container-2.3.0-2.el7ost.noarch
python-swiftclient-2.4.0-1.el7ost.noarch
openstack-swift-object-2.3.0-2.el7ost.noarch
openstack-swift-proxy-2.3.0-2.el7ost.noarch
instack-undercloud-2.1.2-37.el7ost.noarch
openstack-swift-account-2.3.0-2.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-106.el7ost.noarch
openstack-swift-2.3.0-2.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud 7.1 with director.
2. Register the nodes with Sat5 (pointing to 7.3).
3. Update the undercloud
4. Attempt  to update the OC nodes.


Result:

After running some time and actually yum updating the nodes, the deployment fails:
...
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED


Doing some debuging points to SwiftDevicesAndProxyConfig:

[stack@undercloud ~]$ heat resource-list overcloud|grep -v COMPLE
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time         |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| SwiftDevicesAndProxyConfig                | 515b477a-4261-4b10-bb53-4d9adae478eb          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | UPDATE_FAILED   | 2016-01-21T21:53:39Z |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+


Additional debugging shows: resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned



[stack@undercloud ~]$ heat resource-show overcloud SwiftDevicesAndProxyConfig
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                              |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                  |
|                        |   "config_id": "1f069129-cd23-4761-ac8a-769847c8163b"                                                                                                              |
|                        | }                                                                                                                                                                  |
| description            |                                                                                                                                                                    |
| links                  | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud/9799dd92-c319-4878-a605-bda9dd5ea6d0/resources/SwiftDevicesAndProxyConfig (self)      |
|                        | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud/9799dd92-c319-4878-a605-bda9dd5ea6d0 (stack)                                          |
|                        | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud-SwiftDevicesAndProxyConfig-dbbgtzmy7ewg/515b477a-4261-4b10-bb53-4d9adae478eb (nested) |
| logical_resource_id    | SwiftDevicesAndProxyConfig                                                                                                                                         |
| physical_resource_id   | 515b477a-4261-4b10-bb53-4d9adae478eb                                                                                                                               |
| required_by            | ObjectStorageSwiftDeployment                                                                                                                                       |
|                        | ControllerSwiftDeployment                                                                                                                                          |
| resource_name          | SwiftDevicesAndProxyConfig                                                                                                                                         |
| resource_status        | UPDATE_FAILED                                                                                                                                                      |
| resource_status_reason | ValueError: resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned                                                        |
| resource_type          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig                                                                                                                  |
| updated_time           | 2016-01-21T21:53:39Z                                                                                                                                               |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+


Expected result:
completed update.

--- Additional comment from James Slagle on 2016-01-22 07:49:47 EST ---

In 7.2, the resource looked like:

  SwiftDevicesAndProxyConfig:
    type: OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig
    properties:
      controller_swift_devices: {get_attr: [Controller, swift_device]}
      object_store_swift_devices: {get_attr: [ObjectStorage, swift_device]}
      controller_swift_proxy_memcaches: {get_attr: [Controller, swift_proxy_memcache]}

In 7.3, we've added some *_v6 properties (which should always be set):

  SwiftDevicesAndProxyConfig:
    type: OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig
    properties:
      controller_swift_devices: {get_attr: [Controller, swift_device]}
      controller_swift_devices_v6: {get_attr: [Controller, swift_device_v6]}
      object_store_swift_devices: {get_attr: [ObjectStorage, swift_device]}
      object_store_swift_devices_v6: {get_attr: [ObjectStorage, swift_device_v6]}
      controller_swift_proxy_memcaches: {get_attr: [Controller, swift_proxy_memcache]}
      controller_swift_proxy_memcaches_v6: {get_attr: [Controller, swift_proxy_memcache_v6]}

Also in 7.3 the *_v6 attributes are *new* outputs on the Controller resource. So I'm wondering if we're hitting some slight nuance here with the Update logic since this is failing during an update. Do we need to add an explicit depends_on: Controller on the SwiftDevicesAndProxyConfig resource? Would that ensure that Controller is updated first and provides the new outputs (assuming that's even what's triggering the error message)?

--- Additional comment from Zane Bitter on 2016-02-09 13:39:51 EST ---

Thomas has established that this occurs when:

* You update the stack with a child template that adds new mandatory properties
* The update fails for any reason _before_ the point of updating this resource
* You run another update

Comment 6 Alexander Chuzhoy 2016-04-20 20:26:43 UTC
Verified:
Environment:
openstack-heat-common-5.0.1-5.el7ost.noarch

The upgrade from 7.3 -> 8.0 succeeded.
Verified that the patches landed.