Description of problem: Sometimes deletion of stack or image fails, following trace of failed stack Heat-engine.log Controller-3: 2015-09-30 10:40:16.950 2776 INFO heat.engine.resource [-] DELETE: FloatingIPAssociation "vgp2_floating_ip_assoc" [d5c81aa1-7e02-4875-b764-63578d8aae08:dbeda84a-f30b-4e49-986f-6a8c2da4fb5e] Stack "vgp_sriov" [50cb6836-672a-4ceb-9253-3d9331c4a8ad] 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource Traceback (most recent call last): 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 439, in _action_recorder 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource yield 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 843, in delete 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource yield self.action_handler_task(action, *action_args) 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 286, in wrapper 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource step = next(subtask) 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 480, in action_handler_task 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource handler_data = handler(*args) 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/neutron/floatingip.py", line 270, in handle_delete 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource self.client_plugin().ignore_not_found(ex) 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 90, in ignore_not_found 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource raise ex 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource InternalServerError: Request Failed: internal server error while processing your request. 2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource 2015-09-30 10:40:17.057 2776 INFO heat.engine.stack [-] Stack DELETE FAILED (vgp_sriov): Resource DELETE failed: InternalServerError: Request Failed: internal server error while processing your request. controller2 server.log [ochalups@collab-shell 01515783]$ egrep -i 'error|fail' neutron-controller*/var/log/neutron/server.log|grep "09-30 10:4" 2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource self.errorhandler(self, exc, value) 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource raise errorclass, errorvalue 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08') Can the mentioned bug fix targeted for liberty be possibly used as backport in this case? https://bugs.launchpad.net/neutron/+bug/1422504 Can we have a backport to Juno ? Version-Release number of selected component (if applicable): How reproducible: 1 of 20 tries Steps to Reproduce: 1. Delete a stack or image Actual results: 2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource self.errorhandler(self, exc, value) 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource raise errorclass, errorvalue 2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08') Expected results: Stack deleted Additional info:
According stacktrace and launchpad this issue should be reported for Neutron, no?
Yes, neutron. Sorry for wrong component info.
The wrapper was copied into stable/juno. See neutron/db/api.py. So there is no dependency on oslo.db. Am I correct?
(In reply to Ihar Hrachyshka from comment #5) > The wrapper was copied into stable/juno. See neutron/db/api.py. So there is > no dependency on oslo.db. Am I correct? Yes, thanks :) That saves a lot of work.
upstream juno backport submitted https://review.openstack.org/#/c/231985
The code within the release, tested osp6 on rhel7 openstack-neutron-2014.2.3-23.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1950.html
Created attachment 1097699 [details] controller1-heat
For updates to this bug in the future, please see bug #1297946. Additional fixes were required and will be resolved when bug 1297946 is closed.