Bug 1268787 - DBDeadLock Error deleting floating IP
Summary: DBDeadLock Error deleting floating IP
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: async
: 6.0 (Juno)
Assignee: Jakub Libosvar
QA Contact: Ofer Blaut
URL:
Whiteboard:
Depends On:
Blocks: 1297946
TreeView+ depends on / blocked
 
Reported: 2015-10-05 08:53 UTC by Ondrej
Modified: 2019-09-12 09:02 UTC (History)
21 users (show)

Fixed In Version: openstack-neutron-2014.2.3-27.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1297946 (view as bug list)
Environment:
Last Closed: 2016-01-12 20:20:16 UTC
Target Upstream Version:


Attachments (Terms of Use)
controller1-heat (724.33 KB, application/x-bzip)
2015-11-23 14:41 UTC, Ondrej
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1422504 0 None None None Never
OpenStack gerrit 194174 0 None None None Never
OpenStack gerrit 202449 0 None None None Never
OpenStack gerrit 231985 0 None None None Never
Red Hat Product Errata RHBA-2015:1950 0 normal SHIPPED_LIVE openstack-neutron bug fix advisory 2015-10-29 15:38:18 UTC

Description Ondrej 2015-10-05 08:53:37 UTC
Description of problem:

Sometimes deletion of stack or image fails, following trace of failed stack 

Heat-engine.log Controller-3:
2015-09-30 10:40:16.950 2776 INFO heat.engine.resource [-] DELETE: FloatingIPAssociation "vgp2_floating_ip_assoc" [d5c81aa1-7e02-4875-b764-63578d8aae08:dbeda84a-f30b-4e49-986f-6a8c2da4fb5e] Stack "vgp_sriov" [50cb6836-672a-4ceb-9253-3d9331c4a8ad]
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource Traceback (most recent call last):
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 439, in _action_recorder
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     yield
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 843, in delete
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     yield self.action_handler_task(action, *action_args)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 286, in wrapper
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     step = next(subtask)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 480, in action_handler_task
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     handler_data = handler(*args)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/neutron/floatingip.py", line 270, in handle_delete
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     self.client_plugin().ignore_not_found(ex)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 90, in ignore_not_found
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     raise ex
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource InternalServerError: Request Failed: internal server error while processing your request.
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource
2015-09-30 10:40:17.057 2776 INFO heat.engine.stack [-] Stack DELETE FAILED (vgp_sriov): Resource DELETE failed: InternalServerError: Request Failed: internal server error while processing your request.


controller2  server.log
[ochalups@collab-shell 01515783]$ egrep -i 'error|fail' neutron-controller*/var/log/neutron/server.log|grep "09-30 10:4"
2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08')


Can the mentioned bug fix targeted for liberty be possibly used as backport in this case? 

https://bugs.launchpad.net/neutron/+bug/1422504

Can we have a backport to Juno ?



Version-Release number of selected component (if applicable):


How reproducible:
1 of 20 tries

Steps to Reproduce:
1. Delete a stack or image


Actual results:

2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08')

Expected results:

Stack deleted

Additional info:

Comment 2 Sahid Ferdjaoui 2015-10-05 10:38:38 UTC
According stacktrace and launchpad this issue should be reported for Neutron, no?

Comment 3 Ondrej 2015-10-05 10:45:41 UTC
Yes, neutron. Sorry for wrong component info.

Comment 5 Ihar Hrachyshka 2015-10-06 11:26:23 UTC
The wrapper was copied into stable/juno. See neutron/db/api.py. So there is no dependency on oslo.db. Am I correct?

Comment 6 Jakub Libosvar 2015-10-06 12:22:56 UTC
(In reply to Ihar Hrachyshka from comment #5)
> The wrapper was copied into stable/juno. See neutron/db/api.py. So there is
> no dependency on oslo.db. Am I correct?

Yes, thanks :) That saves a lot of work.

Comment 7 Jon Schlueter 2015-10-07 14:26:00 UTC
upstream juno backport submitted

https://review.openstack.org/#/c/231985

Comment 9 Toni Freger 2015-10-20 05:42:43 UTC
The code within the release, tested osp6 on rhel7

openstack-neutron-2014.2.3-23.el7ost.noarch

Comment 12 errata-xmlrpc 2015-10-29 11:39:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1950.html

Comment 15 Ondrej 2015-11-23 14:41:33 UTC
Created attachment 1097699 [details]
controller1-heat

Comment 25 Lon Hohberger 2016-01-12 20:20:16 UTC
For updates to this bug in the future, please see bug #1297946.

Additional fixes were required and will be resolved when bug 1297946 is closed.


Note You need to log in before you can comment on or make changes to this bug.