Bug 1460006 - [OSP8] [HEAT] Resource dependencies not honored on stack update with deleted resources.
[OSP8] [HEAT] Resource dependencies not honored on stack update with deleted ...
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Zane Bitter
Amit Ugol
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-08 15:48 EDT by Harald Jensås
Modified: 2017-07-28 22:11 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Harald Jensås 2017-06-08 15:48:21 EDT
Description of problem:
Heat template contain many resource combinations of PolicyClassifier, PolicyRuleSet, PolycyRule resources. For ICMP Policy, SSH Policy etc.

Example:
"""
resources:
 84458446PolicyClassifier:
    type: OS::GroupBasedPolicy::PolicyClassifier
    properties:
      name: icmp-classifier
      protocol: icmp
      direction: bi
      shared: false

  84458446PolicyRule:
    type: OS::GroupBasedPolicy::PolicyRule
    properties:
      name: icmp-allow-rule
      policy_classifier_id: {get_resource: 84458446PolicyClassifier}
      policy_actions:
        - {get_resource: AllowPolicyAction}
      shared: false

  84458446PolicyRuleSet:
    type: OS::GroupBasedPolicy::PolicyRuleSet
    properties:
      name: icmp-allow-ruleset
      policy_rules:
        - {get_resource: 84458446PolicyRule}
      shared: false
"""

The user removed one such set of resources and did a stack update.

This results in this error:
 "PolicyClassifier 7cb6f7a8-fb58-418f-be29-35144ea09f8d is in use"

This is in the logs:

Jun  7 17:26:57 hostname heat-engine: 2017-06-07 17:26:57.719 4046 INFO heat.engine.resource [-] deleting PolicyClassifier "84458446PolicyClassifier" [7cb6f7a8-fb58-418f-be29-35144ea09f8d] Stack "project1*" [a1b4a653-6e0d-436b-a859-7777f9bed218]
Jun  7 17:26:57 hostname heat-engine: 2017-06-07 17:26:57.793 4046 DEBUG neutronclient.client [-] REQ: curl -i https://hostname.example.com:13696//v2.0/grouppolicy/policy_classifiers/7cb6f7a8-fb58-418f-be29-35144ea09f8d.json -X DELETE -H "User-Agent: python-neutronclient" -H "X-Auth-Token: 54c4de2a85d64afba6c70ddb732b5c0c" http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:137
Jun  7 17:26:57 hostname heat-engine: 2017-06-07 17:26:57.904 4046 DEBUG neutronclient.client [-] RESP: 409 {'date': 'Wed, 07 Jun 2017 13:26:57 GMT', 'connection': 'keep-alive', 'content-type': 'application/json; charset=UTF-8', 'content-length': '173', 'x-openstack-request-id': 'req-a506cc5a-34fc-4af6-afdf-f7c331862100'} {"NeutronError": {"message": "Unable to complete operation, PolicyClassifier 7cb6f7a8-fb58-418f-be29-35144ea09f8d is in use", "type": "PolicyClassifierInUse", "detail": ""}} http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:146
Jun  7 17:26:57 hostname heat-engine: 2017-06-07 17:26:57.904 4046 DEBUG gbpclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Unable to complete operation, PolicyClassifier 7cb6f7a8-fb58-418f-be29-35144ea09f8d is in use", "type": "PolicyClassifierInUse", "detail": ""}} _handle_fault_response /usr/lib/python2.7/site-packages/gbpclient/v2_0/client.py:716
Jun  7 17:26:57 hostname heat-engine: 2017-06-07 17:26:57.905 4046 INFO heat.engine.resource [-] DELETE: PolicyClassifier "84458446PolicyClassifier" [7cb6f7a8-fb58-418f-be29-35144ea09f8d] Stack "project1*" [a1b4a653-6e0d-436b-a859-7777f9bed218]
Jun  7 17:26:57 hostname heat-engine: 2017-06-07 17:26:57.905 4046 ERROR heat.engine.resource Conflict: Unable to complete operation, PolicyClassifier 7cb6f7a8-fb58-418f-be29-35144ea09f8d is in use
Jun  7 17:26:58 hostname heat-engine: 2017-06-07 17:26:58.018 4046 INFO heat.engine.stack [-] Stack UPDATE FAILED (project1): Conflict: resources.84458446PolicyClassifier: Unable to complete operation, PolicyClassifier 7cb6f7a8-fb58-418f-be29-35144ea09f8d is in use


Looking at resources in the stack we found the dependent resource:

$ heat resource-show project1 84458446PolicyClassifier
+------------------------+----------------------------------------------------------------
| Property               | Value
+------------------------+----------------------------------------------------------------
| logical_resource_id    | 84458446PolicyClassifier
| physical_resource_id   | 43160099-7183-480f-a78f-cec61ecd947d
| required_by            | 84458446PolicyRule                <----- required by
| resource_name          | 84458446PolicyClassifier
| resource_type          | OS::GroupBasedPolicy::PolicyClassifier

$ heat resource-show project1 84458446PolicyRule
+------------------------+--------------------------------------------------------------------------------------------
| Property               | Value                                                                                                            
+------------------------+--------------------------------------------------------------------------------------------
| attributes             | {
|                        |   "policy_classifier_id": "7cb6f7a8-fb58-418f-be29-35144ea09f8d",  <----- classifier in use
| logical_resource_id    | 84458446PolicyRule
| physical_resource_id   | 1452937f-f7e9-4a7a-8093-c87b21973d7c
| required_by            |                                          <----- required by none
| resource_name          | 84458446PolicyRule
| resource_type          | OS::GroupBasedPolicy::PolicyRule





Version-Release number of selected component (if applicable):
openstack-heat-api-5.0.1-9.el7ost.noarch                    Sat Dec  3 00:49:50 2016
openstack-heat-api-cfn-5.0.1-9.el7ost.noarch                Sat Dec  3 00:49:51 2016
openstack-heat-api-cloudwatch-5.0.1-9.el7ost.noarch         Sat Dec  3 00:49:51 2016
openstack-heat-common-5.0.1-9.el7ost.noarch                 Sat Dec  3 00:49:04 2016
openstack-heat-engine-5.0.1-9.el7ost.noarch                 Sat Dec  3 00:49:50 2016
openstack-heat-gbp-2015.2.2-71.el7.noarch                   Mon Apr  3 18:04:12 2017


Actual results:
Stack update fails with: "PolicyClassifier 7cb6f7a8-fb58-418f-be29-35144ea09f8d is in use" because resource dependencies are not honored. The resources are deleted in the wrong order.

Expected results:
Since both OS::GroupBasedPolicy::PolicyRule and OS::GroupBasedPolicy::PolicyRuleSet resources refer to other resources with {get_resource: ResourceName} Heat should be able to resolve the dependencies and delete the resources in the correct order.
Comment 1 Zane Bitter 2017-06-08 16:33:54 EDT
We do not ship, and have never shipped, OS::GroupBasedPolicy:: resources in Heat (not even upstream), and I have never seen the source for these resources. There is no openstack-heat-gbp package in RHOS. There is no way we can support this.

Explicit dependencies like this not being honoured is something that we never see in Heat - that part of the code is extremely robust and well-tested - so I suspect problems with the unsupported plugins.
Comment 2 Zane Bitter 2017-06-08 17:34:08 EDT
Looking at the logs, it appears to be failing when deleting a backup resource (i.e. the original resource has been replaced, and then a failure occurs subsequently so that downstream resources are not updated to point to the replacement, and then rollback either fails or is disabled). Usually this will just work correctly on the next update, since we swap the backup resource and the current resource before trying to delete the backup stack - see https://launchpadlibrarian.net/175881721/reason.pdf

The latest RHOS 8 Heat version is 5.0.3-2, so the first thing to try is to update to that.
Comment 3 Harald Jensås 2017-06-09 06:37:52 EDT
Thanks Zane for looking at this.

The OS::GroupBasedPolicy:: resources is indeed not shipped by Red Hat, these are shipped by Cisco as part of their ACI/APIC integration. We have already a case opened by the customer with the provider of the openstack-heat-gbp package to investigate the third party plugins.

The GroupBasedPolicy plug-ins code is:
https://github.com/openstack/group-based-policy-automation/tree/master/gbpautomation/heat


Doing another update did not help in this case.
I will suggest the customer to update Heat.

Note You need to log in before you can comment on or make changes to this bug.