Bug 1031034 (heat-bp-support-retry-with-idempotency)

Summary: [RFE][heat]: Support API retry function with Idempotency in creating/updating a stack
Product: Red Hat OpenStack Reporter: Stephen Gordon <sgordon>
Component: openstack-heatAssignee: Steve Baker <sbaker>
Status: CLOSED WONTFIX QA Contact: Amit Ugol <augol>
Severity: unspecified Docs Contact:
Priority: high    
Version: unspecifiedCC: markmc, sbaker, shardy, srevivo, zbitter
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: https://blueprints.launchpad.net/heat/+spec/support-retry-with-idempotency
Whiteboard: upstream_milestone_none upstream_status_not-started upstream_definition_drafting
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-04 21:19:30 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1031047    

Description Stephen Gordon 2013-11-15 14:04:59 UTC 
Cloned from launchpad blueprint https://blueprints.launchpad.net/heat/+spec/support-retry-with-idempotency.

Description:

Currently Heat doesn't have a retry function in creating/updating a stack.
In case of API request failure, Heat would change stack status to "XXX_FAILURE" .
(Or start rollback process if the rollback flag was true.)

However, I think there are some circumstances/scenarios where retry is appropriate.
(i.e. 503 response or timeout due to server failover)

On the other hand, however, a retry function may result in creating duplicate resources when Heat couldn't get response from nova and other APIs.
In order to avoid this, I believe leveraging idempotency that is discussed in nova could be a solution.

https://blueprints.launchpad.net/nova/+spec/idempotentcy-client-token

Our idea for implementing a retry function to Heat is:
Heat to retry POST requests with a ClientToken header to avoid creating duplicate resources, until it reaches a retry limit.
Heat to retry PUT/GET requests until it gets 2xx response or it reaches a retry limit. ClientTokens are not used in this case.
Heat to retry DELETE requests until it gets 2xx response or 404 response or it reaches a retry limit. ClientTokens are not used in this case.

Note:
Some of pythonclients have retry functions, but they cannot be utilized because Heat has to judge the necessity of retry by itself.


Specification URL (additional information):

None
Comment 2 Zane Bitter 2016-05-04 21:19:30 UTC 
We have retries, but idempotency relies on changes to the Nova API which never landed. Closing this bug as there's no prospect of it happening upstream any time soon.