Bug 1310879 - rhel-osp-director: Overcloud update from 7.2-> 7.3 fails "resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned "
rhel-osp-director: Overcloud update from 7.2-> 7.3 fails "resources.SwiftDevi...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
urgent Severity unspecified
: ga
: 8.0 (Liberty)
Assigned To: Zane Bitter
Omri Hochman
: TestOnly, ZStream
Depends On: 1300847
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-22 16:19 EST by Zane Bitter
Modified: 2016-04-28 09:51 EDT (History)
16 users (show)

See Also:
Fixed In Version: openstack-heat-5.0.1-2.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1300847
Environment:
Last Closed: 2016-04-28 09:51:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1543685 None None None 2016-02-22 16:19 EST
OpenStack gerrit 278828 None None None 2016-02-22 16:19 EST

  None (edit)
Description Zane Bitter 2016-02-22 16:19:46 EST
+++ This bug was initially created as a clone of Bug #1300847 +++

rhel-osp-director: Overcloud update from 7.2-> 7.3 fails: "resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned"


Environment:
openstack-swift-container-2.3.0-2.el7ost.noarch
python-swiftclient-2.4.0-1.el7ost.noarch
openstack-swift-object-2.3.0-2.el7ost.noarch
openstack-swift-proxy-2.3.0-2.el7ost.noarch
instack-undercloud-2.1.2-37.el7ost.noarch
openstack-swift-account-2.3.0-2.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-106.el7ost.noarch
openstack-swift-2.3.0-2.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud 7.1 with director.
2. Register the nodes with Sat5 (pointing to 7.3).
3. Update the undercloud
4. Attempt  to update the OC nodes.


Result:

After running some time and actually yum updating the nodes, the deployment fails:
...
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED


Doing some debuging points to SwiftDevicesAndProxyConfig:

[stack@undercloud ~]$ heat resource-list overcloud|grep -v COMPLE
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time         |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+
| SwiftDevicesAndProxyConfig                | 515b477a-4261-4b10-bb53-4d9adae478eb          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | UPDATE_FAILED   | 2016-01-21T21:53:39Z |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+


Additional debugging shows: resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned



[stack@undercloud ~]$ heat resource-show overcloud SwiftDevicesAndProxyConfig
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                              |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                  |
|                        |   "config_id": "1f069129-cd23-4761-ac8a-769847c8163b"                                                                                                              |
|                        | }                                                                                                                                                                  |
| description            |                                                                                                                                                                    |
| links                  | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud/9799dd92-c319-4878-a605-bda9dd5ea6d0/resources/SwiftDevicesAndProxyConfig (self)      |
|                        | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud/9799dd92-c319-4878-a605-bda9dd5ea6d0 (stack)                                          |
|                        | http://192.168.0.1:8004/v1/4484871019d94a2aa5933630632d3f47/stacks/overcloud-SwiftDevicesAndProxyConfig-dbbgtzmy7ewg/515b477a-4261-4b10-bb53-4d9adae478eb (nested) |
| logical_resource_id    | SwiftDevicesAndProxyConfig                                                                                                                                         |
| physical_resource_id   | 515b477a-4261-4b10-bb53-4d9adae478eb                                                                                                                               |
| required_by            | ObjectStorageSwiftDeployment                                                                                                                                       |
|                        | ControllerSwiftDeployment                                                                                                                                          |
| resource_name          | SwiftDevicesAndProxyConfig                                                                                                                                         |
| resource_status        | UPDATE_FAILED                                                                                                                                                      |
| resource_status_reason | ValueError: resources.SwiftDevicesAndProxyConfig: Property controller_swift_proxy_memcaches_v6 not assigned                                                        |
| resource_type          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig                                                                                                                  |
| updated_time           | 2016-01-21T21:53:39Z                                                                                                                                               |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+


Expected result:
completed update.

--- Additional comment from James Slagle on 2016-01-22 07:49:47 EST ---

In 7.2, the resource looked like:

  SwiftDevicesAndProxyConfig:
    type: OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig
    properties:
      controller_swift_devices: {get_attr: [Controller, swift_device]}
      object_store_swift_devices: {get_attr: [ObjectStorage, swift_device]}
      controller_swift_proxy_memcaches: {get_attr: [Controller, swift_proxy_memcache]}

In 7.3, we've added some *_v6 properties (which should always be set):

  SwiftDevicesAndProxyConfig:
    type: OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig
    properties:
      controller_swift_devices: {get_attr: [Controller, swift_device]}
      controller_swift_devices_v6: {get_attr: [Controller, swift_device_v6]}
      object_store_swift_devices: {get_attr: [ObjectStorage, swift_device]}
      object_store_swift_devices_v6: {get_attr: [ObjectStorage, swift_device_v6]}
      controller_swift_proxy_memcaches: {get_attr: [Controller, swift_proxy_memcache]}
      controller_swift_proxy_memcaches_v6: {get_attr: [Controller, swift_proxy_memcache_v6]}

Also in 7.3 the *_v6 attributes are *new* outputs on the Controller resource. So I'm wondering if we're hitting some slight nuance here with the Update logic since this is failing during an update. Do we need to add an explicit depends_on: Controller on the SwiftDevicesAndProxyConfig resource? Would that ensure that Controller is updated first and provides the new outputs (assuming that's even what's triggering the error message)?

--- Additional comment from Zane Bitter on 2016-02-09 13:39:51 EST ---

Thomas has established that this occurs when:

* You update the stack with a child template that adds new mandatory properties
* The update fails for any reason _before_ the point of updating this resource
* You run another update
Comment 6 Alexander Chuzhoy 2016-04-20 16:26:43 EDT
Verified:
Environment:
openstack-heat-common-5.0.1-5.el7ost.noarch

The upgrade from 7.3 -> 8.0 succeeded.
Verified that the patches landed.

Note You need to log in before you can comment on or make changes to this bug.