Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1626367

Summary: Ignore spurious nested stack locks in convergence
Product: Red Hat OpenStack Reporter: Pierre Amadio <pamadio>
Component: openstack-heatAssignee: Zane Bitter <zbitter>
Status: CLOSED NEXTRELEASE QA Contact: Ronnie Rasouli <rrasouli>
Severity: medium Docs Contact:
Priority: medium    
Version: 11.0 (Ocata)CC: aschultz, cylopez, mburns, pmorey, sbaker, shardy, srevivo, therve, zbitter
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 11.0 (Ocata)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-heat-8.0.6-4.el7ost Doc Type: Bug Fix
Doc Text:
Cause: When convergence is enabled for a stack, the "stack check" operation uses a different type of locking (stack-level instead of resource-level) from regular operations such as "stack update" or "stack delete". However, when processing nested stacks they are not considered complete until any stack-level lock is released. Consequence: Any stale locks left behind in a nested stack by a stack check will cause the parent stack to time out when updating or deleting the nested stack. Fix: Ignore the stack-level locks when checking the completion of any nested stack operation (like update or delete in convergence stacks) that uses only resource-level locks. Result: Any stray locks left behind in the database will not prevent further operations on the nested stack.
 Story Points: ---
Clone Of:
: 1632295 (view as bug list) Environment:
Last Closed: 2019-01-23 19:31:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1632295, 1632313    

Description Pierre Amadio 2018-09-07 07:42:56 UTC 
This is a bug to request a backport of this change on RHOSP11:

https://review.openstack.org/#/c/598354/

Several operations (e.g. stack check) are yet to be converted to convergence-style workflows, and still create locks. While we try to always remove dead locks, it's possible that if one of these gets left behind we won't notice, since convergence doesn't actually use stack locks for most regular operations. When doing _check_status_complete() on a nested stack resource, we wait for the operation to release the nested stack's lock before reporting the resource completed, to ensure that for legacy operations the nested stack is ready to perform another action on as soon as the resource is complete. For convergence stacks this is an unnecessary DB call, and it can lead to resources never completing if a stray lock happens to be left in the database. Only check the nested stack's stack lock for operations where we are not taking a resource lock. This corresponds exactly to legacy-style operations.
Comment 4 Cyril Lopez 2018-09-14 10:34:52 UTC 
I was able to reproduce 

2018-09-14 06:25:25.246 3527 INFO heat.engine.service [req-d85efee4-3729-4a57-b54f-f7ca2e24bd80 - admin - - -] Deleting stack mastack-asg-na6iipkowaov
2018-09-14 06:25:25.294 3527 INFO heat.engine.stack [req-d85efee4-3729-4a57-b54f-f7ca2e24bd80 - admin - - -] Stack DELETE FAILED (mastack-asg-na6iipkowaov): Stack DELETE cancelled
2018-09-14 06:25:25.839 3525 INFO heat.engine.resource [req-3fc8001d-9692-4563-b82f-bcc44b0b055a - admin - - -] DELETE: AutoScalingResourceGroup "asg" [a22f3ac5-6008-4203-a325-fedfd10d097e] Stack "mastack" [942710a9-e86b-4d4b-af5c-18791e96595d]
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource Traceback (most recent call last):
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 793, in _action_recorder
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource     yield
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1729, in delete
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource     *action_args)
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 352, in wrapper
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource     step = next(subtask)
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 846, in action_handler_task
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource     done = check(handler_data)
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 553, in check_delete_complete
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource     return self._check_status_complete(self.DELETE)
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 419, in _check_status_complete
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource     action=action)
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource ResourceFailure: resources.asg: Stack DELETE cancelled
2018-09-14 06:25:25.839 3525 ERROR heat.engine.resource 
2018-09-14 06:25:25.919 3525 INFO heat.engine.stack [req-3fc8001d-9692-4563-b82f-bcc44b0b055a - admin - - -] Stack DELETE FAILED (mastack): Resource DELETE failed: resources.asg: Stack DELETE cancelled
2018-09-14 06:25:30.333 3527 INFO heat.engine.worker [req-d85efee4-3729-4a57-b54f-f7ca2e24bd80 - admin - - -] [mastack-asg-na6iipkowaov(a22f3ac5-6008-4203-a325-fedfd10d097e)] Stopped all active workers for stack DELETE
2018-09-14 06:25:30.377 3527 INFO heat.engine.stack [req-b6020d84-b5a1-463a-9106-ca179c5617f6 - - - - -] Stack DELETE IN_PROGRESS (mastack-asg-na6iipkowaov): Stack DELETE started
2018-09-14 06:25:30.417 3527 INFO heat.engine.stack [req-b6020d84-b5a1-463a-9106-ca179c5617f6 - - - - -] convergence_dependencies: (Dependencies([((86, False), None)]),)
2018-09-14 06:25:30.426 3527 INFO heat.engine.stack [req-b6020d84-b5a1-463a-9106-ca179c5617f6 - - - - -] Triggering resource 86 for cleanup
2018-09-14 06:25:30.520 3527 INFO heat.engine.resource [req-d85efee4-3729-4a57-b54f-f7ca2e24bd80 - admin - - -] deleting (<heat.engine.resources.openstack.heat.resource_group.ResourceGroup object at 0x7fc56a2f4f50>,)
2018-09-14 06:25:30.540 3527 INFO heat.engine.resource [req-d85efee4-3729-4a57-b54f-f7ca2e24bd80 - admin - - -] delete ResourceGroup "bhmrulytdtlw" [594cdbcc-16fd-49e8-b803-b9ba927111cc] Stack "mastack-asg-na6iipkowaov" [a22f3ac5-6008-4203-a325-fedfd10d097e] attempt 1

With apply the upstream patch, I was able to delete it in 2 times (2 call of openstack stack delete)

2018-09-14 06:32:01.061 10557 ERROR heat.engine.worker [req-a465ab5f-8ef9-44b2-b2b2-82f749955492 - admin - - -] Failed to stop all workers of stack mastack-as
g-na6iipkowaov , stack cancel not complete
2018-09-14 06:32:01.118 10557 INFO heat.engine.stack [req-a590bf23-0d9f-4b31-af66-7d824d7c7aef - - - - -] Stack DELETE IN_PROGRESS (mastack-asg-na6iipkowaov): Stack DELETE started
2018-09-14 06:32:01.163 10557 INFO heat.engine.stack [req-a590bf23-0d9f-4b31-af66-7d824d7c7aef - - - - -] convergence_dependencies: (Dependencies([((86, False), None)]),)
2018-09-14 06:32:01.173 10557 INFO heat.engine.stack [req-a590bf23-0d9f-4b31-af66-7d824d7c7aef - - - - -] Triggering resource 86 for cleanup
2018-09-14 06:32:01.283 10556 INFO heat.engine.resource [req-a465ab5f-8ef9-44b2-b2b2-82f749955492 - admin - - -] Resource ResourceGroup "bhmrulytdtlw" [594cdbcc-16fd-49e8-b803-b9ba927111cc] 
Stack "mastack-asg-na6iipkowaov" [a22f3ac5-6008-4203-a325-fedfd10d097e] is locked for update; deferring
2018-09-14 06:32:03.481 10555 INFO heat.engine.resource [req-a465ab5f-8ef9-44b2-b2b2-82f749955492 - admin - - -] deleting (<heat.engine.resources.openstack.heat.resource_group.ResourceGroup 
object at 0x7fe55b9c2250>,)
2018-09-14 06:32:03.506 10555 INFO heat.engine.resource [req-a465ab5f-8ef9-44b2-b2b2-82f749955492 - admin - - -] delete ResourceGroup "bhmrulytdtlw" [594cdbcc-16fd-49e8-b803-b9ba927111cc] St
ack "mastack-asg-na6iipkowaov" [a22f3ac5-6008-4203-a325-fedfd10d097e] attempt 1
2018-09-14 06:32:03.617 10555 INFO heat.engine.stack [req-a465ab5f-8ef9-44b2-b2b2-82f749955492 - admin - - -] [mastack-asg-na6iipkowaov(a22f3ac5-6008-4203-a325-fedfd10d097e)] update traversa
l 9970d78c-666d-40ba-8d9f-627905393c9b complete
2018-09-14 06:32:03.661 10555 INFO heat.engine.stack [req-a465ab5f-8ef9-44b2-b2b2-82f749955492 - admin - - -] Stack DELETE COMPLETE (mastack-asg-na6iipkowaov): Stack DELETE completed success
fully
2018-09-14 06:32:29.466 10556 INFO heat.engine.service [req-ce7bc708-bd36-44c0-a02d-248348019508 - admin - - -] Deleting stack mastack
2018-09-14 06:32:29.475 10556 INFO heat.engine.worker [req-ce7bc708-bd36-44c0-a02d-248348019508 - admin - - -] [mastack(942710a9-e86b-4d4b-af5c-18791e96595d)] Stopped all active workers for stack DELETE
2018-09-14 06:32:29.526 10556 INFO heat.engine.stack [req-8c5f5b29-ce21-4769-9c88-3a7ea97fe742 - - - - -] Stack DELETE IN_PROGRESS (mastack): Stack DELETE started
2018-09-14 10:32:29Z [mastack]: DELETE_IN_PROGRESS  Stack DELETE started
2018-09-14 06:32:29.567 10556 INFO heat.engine.stack [req-8c5f5b29-ce21-4769-9c88-3a7ea97fe742 - - - - -] convergence_dependencies: (Dependencies([((83, False), None)]),)
2018-09-14 06:32:29.576 10556 INFO heat.engine.stack [req-8c5f5b29-ce21-4769-9c88-3a7ea97fe742 - - - - -] Triggering resource 83 for cleanup
2018-09-14 06:32:29.665 10557 INFO heat.engine.resource [req-ce7bc708-bd36-44c0-a02d-248348019508 - admin - - -] deleting (<heat.engine.resources.openstack.heat.autoscaling_group.AutoScalingResourceGroup object at 0x7fe55b9c2090>,)
2018-09-14 06:32:29.688 10557 INFO heat.engine.resource [req-ce7bc708-bd36-44c0-a02d-248348019508 - admin - - -] delete AutoScalingResourceGroup "asg" [a22f3ac5-6008-4203-a325-fedfd10d097e] Stack "mastack" [942710a9-e86b-4d4b-af5c-18791e96595d] attempt 1
2018-09-14 06:32:29.789 10557 INFO heat.engine.stack [req-ce7bc708-bd36-44c0-a02d-248348019508 - admin - - -] [mastack(942710a9-e86b-4d4b-af5c-18791e96595d)] update traversal 90c1224e-8c31-4f8b-93ac-bff8b139b650 complete
2018-09-14 06:32:29.806 10557 INFO heat.engine.stack [req-ce7bc708-bd36-44c0-a02d-248348019508 - admin - - -] Stack DELETE COMPLETE (mastack): Stack DELETE completed successfully