Bug 1268787

Summary: DBDeadLock Error deleting floating IP
Product: Red Hat OpenStack Reporter: Ondrej <ochalups>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Ofer Blaut <oblaut>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 6.0 (Juno)CC: amuller, berrange, chrisw, dasmith, eglynn, ihrachys, jlibosva, jschluet, kchamart, mschuppe, ndipanov, nyechiel, ochalups, sbauza, sferdjao, sgordon, sknauss, tfreger, vromanso, yeylon
Target Milestone: asyncKeywords: Reopened, ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-2014.2.3-27.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1297946 (view as bug list) Environment:
Last Closed: 2016-01-12 20:20:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1297946    
Attachments:
Description Flags
controller1-heat none

Description Ondrej 2015-10-05 08:53:37 UTC 
Description of problem:

Sometimes deletion of stack or image fails, following trace of failed stack 

Heat-engine.log Controller-3:
2015-09-30 10:40:16.950 2776 INFO heat.engine.resource [-] DELETE: FloatingIPAssociation "vgp2_floating_ip_assoc" [d5c81aa1-7e02-4875-b764-63578d8aae08:dbeda84a-f30b-4e49-986f-6a8c2da4fb5e] Stack "vgp_sriov" [50cb6836-672a-4ceb-9253-3d9331c4a8ad]
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource Traceback (most recent call last):
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 439, in _action_recorder
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     yield
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 843, in delete
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     yield self.action_handler_task(action, *action_args)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 286, in wrapper
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     step = next(subtask)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 480, in action_handler_task
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     handler_data = handler(*args)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/neutron/floatingip.py", line 270, in handle_delete
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     self.client_plugin().ignore_not_found(ex)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 90, in ignore_not_found
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     raise ex
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource InternalServerError: Request Failed: internal server error while processing your request.
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource
2015-09-30 10:40:17.057 2776 INFO heat.engine.stack [-] Stack DELETE FAILED (vgp_sriov): Resource DELETE failed: InternalServerError: Request Failed: internal server error while processing your request.


controller2  server.log
[ochalups@collab-shell 01515783]$ egrep -i 'error|fail' neutron-controller*/var/log/neutron/server.log|grep "09-30 10:4"
2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08')


Can the mentioned bug fix targeted for liberty be possibly used as backport in this case? 

https://bugs.launchpad.net/neutron/+bug/1422504

Can we have a backport to Juno ?



Version-Release number of selected component (if applicable):


How reproducible:
1 of 20 tries

Steps to Reproduce:
1. Delete a stack or image


Actual results:

2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08')

Expected results:

Stack deleted

Additional info:
Comment 2 Sahid Ferdjaoui 2015-10-05 10:38:38 UTC 
According stacktrace and launchpad this issue should be reported for Neutron, no?
Comment 3 Ondrej 2015-10-05 10:45:41 UTC 
Yes, neutron. Sorry for wrong component info.
Comment 5 Ihar Hrachyshka 2015-10-06 11:26:23 UTC 
The wrapper was copied into stable/juno. See neutron/db/api.py. So there is no dependency on oslo.db. Am I correct?
Comment 6 Jakub Libosvar 2015-10-06 12:22:56 UTC 
(In reply to Ihar Hrachyshka from comment #5)
> The wrapper was copied into stable/juno. See neutron/db/api.py. So there is
> no dependency on oslo.db. Am I correct?

Yes, thanks :) That saves a lot of work.
Comment 7 Jon Schlueter 2015-10-07 14:26:00 UTC 
upstream juno backport submitted

https://review.openstack.org/#/c/231985
Comment 9 Toni Freger 2015-10-20 05:42:43 UTC 
The code within the release, tested osp6 on rhel7

openstack-neutron-2014.2.3-23.el7ost.noarch
Comment 12 errata-xmlrpc 2015-10-29 11:39:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1950.html
Comment 15 Ondrej 2015-11-23 14:41:33 UTC 
Created attachment 1097699 [details]
controller1-heat
Comment 25 Lon Hohberger 2016-01-12 20:20:16 UTC 
For updates to this bug in the future, please see bug #1297946.

Additional fixes were required and will be resolved when bug 1297946 is closed.