Bug 1268787 - DBDeadLock Error deleting floating IP
DBDeadLock Error deleting floating IP
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
6.0 (Juno)
Unspecified Unspecified
medium Severity unspecified
: async
: 6.0 (Juno)
Assigned To: Jakub Libosvar
Ofer Blaut
: Reopened, ZStream
Depends On:
Blocks: 1297946
  Show dependency treegraph
 
Reported: 2015-10-05 04:53 EDT by Ondrej
Modified: 2016-09-15 07:10 EDT (History)
22 users (show)

See Also:
Fixed In Version: openstack-neutron-2014.2.3-27.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1297946 (view as bug list)
Environment:
Last Closed: 2016-01-12 15:20:16 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
controller1-heat (724.33 KB, application/x-bzip)
2015-11-23 09:41 EST, Ondrej
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1422504 None None None Never
OpenStack gerrit 194174 None None None Never
OpenStack gerrit 202449 None None None Never
OpenStack gerrit 231985 None None None Never

  None (edit)
Description Ondrej 2015-10-05 04:53:37 EDT
Description of problem:

Sometimes deletion of stack or image fails, following trace of failed stack 

Heat-engine.log Controller-3:
2015-09-30 10:40:16.950 2776 INFO heat.engine.resource [-] DELETE: FloatingIPAssociation "vgp2_floating_ip_assoc" [d5c81aa1-7e02-4875-b764-63578d8aae08:dbeda84a-f30b-4e49-986f-6a8c2da4fb5e] Stack "vgp_sriov" [50cb6836-672a-4ceb-9253-3d9331c4a8ad]
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource Traceback (most recent call last):
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 439, in _action_recorder
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     yield
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 843, in delete
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     yield self.action_handler_task(action, *action_args)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 286, in wrapper
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     step = next(subtask)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 480, in action_handler_task
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     handler_data = handler(*args)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/neutron/floatingip.py", line 270, in handle_delete
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     self.client_plugin().ignore_not_found(ex)
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 90, in ignore_not_found
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource     raise ex
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource InternalServerError: Request Failed: internal server error while processing your request.
2015-09-30 10:40:16.950 2776 TRACE heat.engine.resource
2015-09-30 10:40:17.057 2776 INFO heat.engine.stack [-] Stack DELETE FAILED (vgp_sriov): Resource DELETE failed: InternalServerError: Request Failed: internal server error while processing your request.


controller2  server.log
[ochalups@collab-shell 01515783]$ egrep -i 'error|fail' neutron-controller*/var/log/neutron/server.log|grep "09-30 10:4"
2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08')


Can the mentioned bug fix targeted for liberty be possibly used as backport in this case? 

https://bugs.launchpad.net/neutron/+bug/1422504

Can we have a backport to Juno ?



Version-Release number of selected component (if applicable):


How reproducible:
1 of 20 tries

Steps to Reproduce:
1. Delete a stack or image


Actual results:

2015-09-30 10:40:16.943 25588 ERROR neutron.api.v2.resource [req-99ad3fd7-510e-46a1-96d0-8ff92d509f97 None] update failed
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     self.errorhandler(self, exc, value)
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource     raise errorclass, errorvalue
2015-09-30 10:40:16.943 25588 TRACE neutron.api.v2.resource DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET fixed_port_id=%s, fixed_ip_address=%s, router_id=%s, last_known_router_id=%s WHERE floatingips.id = %s' (None, None, None, '6b32bd0e-98cd-4689-9b0f-f318f2cba83d', 'd5c81aa1-7e02-4875-b764-63578d8aae08')

Expected results:

Stack deleted

Additional info:
Comment 2 Sahid Ferdjaoui 2015-10-05 06:38:38 EDT
According stacktrace and launchpad this issue should be reported for Neutron, no?
Comment 3 Ondrej 2015-10-05 06:45:41 EDT
Yes, neutron. Sorry for wrong component info.
Comment 5 Ihar Hrachyshka 2015-10-06 07:26:23 EDT
The wrapper was copied into stable/juno. See neutron/db/api.py. So there is no dependency on oslo.db. Am I correct?
Comment 6 Jakub Libosvar 2015-10-06 08:22:56 EDT
(In reply to Ihar Hrachyshka from comment #5)
> The wrapper was copied into stable/juno. See neutron/db/api.py. So there is
> no dependency on oslo.db. Am I correct?

Yes, thanks :) That saves a lot of work.
Comment 7 Jon Schlueter 2015-10-07 10:26:00 EDT
upstream juno backport submitted

https://review.openstack.org/#/c/231985
Comment 9 Toni Freger 2015-10-20 01:42:43 EDT
The code within the release, tested osp6 on rhel7

openstack-neutron-2014.2.3-23.el7ost.noarch
Comment 12 errata-xmlrpc 2015-10-29 07:39:48 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1950.html
Comment 15 Ondrej 2015-11-23 09:41 EST
Created attachment 1097699 [details]
controller1-heat
Comment 25 Lon Hohberger 2016-01-12 15:20:16 EST
For updates to this bug in the future, please see bug #1297946.

Additional fixes were required and will be resolved when bug 1297946 is closed.

Note You need to log in before you can comment on or make changes to this bug.