1117818 – DB locking errors when deleting multiple instances that have floating IPs associated to them.

Bug 1117818 - DB locking errors when deleting multiple instances that have floating IPs associated to them.

Summary: DB locking errors when deleting multiple instances that have floating IPs ass...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	5.0 (RHEL 6)
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	5.0 (RHEL 6)
Assignee:	Ihar Hrachyshka
QA Contact:	Ofer Blaut
Docs Contact:
URL:
Whiteboard:
Depends On:	1109577
Blocks:
TreeView+	depends on / blocked

Reported:	2014-07-09 12:54 UTC by Ihar Hrachyshka
Modified:	2018-12-06 17:15 UTC (History)
CC List:	12 users (show)
Fixed In Version:	openstack-neutron-2014.1-37.el6ost
Doc Type:	Bug Fix
Doc Text:	When deleting a port, under some circumstances a deadlock occurred when two inter-dependent threads attempted to access the same database row. This resulted in the database library returning an OperationalError after a delay of approximately 50 seconds. This has been fixed by moving notifications which could cause this type of conflict to outside the database transaction. Now, these types deadlock do not occur.
Clone Of:	1109577
Environment:
Last Closed:	2014-07-28 18:58:50 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1330955	None	None	None	Never
OpenStack gerrit	100934	None	None	None	Never
OpenStack gerrit	105392	None	None	None	Never
Red Hat Product Errata	RHEA-2014:0953	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform Enhancement - Networking	2014-07-28 22:56:42 UTC

Comment 1 Ihar Hrachyshka 2014-07-09 13:01:43 UTC

On related note, Red Hat has started to push upstream to switch to another MySQL client library that should not show those and similar deadlocks: https://review.openstack.org/#/c/104905/

Comment 5 Ihar Hrachyshka 2014-07-18 10:36:56 UTC

@Lee, indeed. I've changed Target to RHEL 6.

Comment 8 Ofer Blaut 2014-07-21 12:30:50 UTC

Verified 

[root@puma05 ~]# rpm -qa | grep neutron
python-neutronclient-2.3.4-2.el6ost.noarch
openstack-neutron-2014.1.1-4.el6ost.noarch
python-neutron-2014.1.1-4.el6ost.noarch
openstack-neutron-metering-agent-2014.1.1-4.el6ost.noarch
openstack-neutron-openvswitch-2014.1.1-4.el6ost.noarch


[root@puma04 ~(keystone_admin_tenant1)]$neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| 0a4462a2-aafe-4bbd-a0cb-a88a1222b112 | 21.0.0.18        | 10.35.180.26        | 6cdccb78-de57-41be-94c4-9e01693acc79 |
| 0c7eedde-bd3e-4df4-89c1-d40f640999e8 | 21.0.0.42        | 10.35.180.50        | c332c3c4-dca9-4d4c-991b-4f8a3f42dbe2 |
| 176515be-ae2e-4f78-ae13-04d1502d1431 | 21.0.0.27        | 10.35.180.35        | a75ea0bc-e4a4-45e4-9450-cbd090053edf |
| 19adb669-0d85-46f8-8535-e0b9b02e499c | 21.0.0.17        | 10.35.180.25        | 995ba35a-d4fe-4ed2-a770-25aab1dba46b |
| 1c0c1999-8c53-46d3-80ec-e0cd95a3af88 | 21.0.0.19        | 10.35.180.27        | 61a3d766-6de9-49dc-bdc9-f382ff70cfda |
| 1c5590f6-5745-41a8-b2c9-8e7d085cf997 | 21.0.0.24        | 10.35.180.32        | 458a936f-8472-40da-96d3-8bb61b0a55d8 |
| 1d88c18e-87de-4815-b984-e09584551b7b | 21.0.0.16        | 10.35.180.24        | c42cce07-bf8c-45a7-b83d-f2ab329744de |
| 20cdfe3b-5616-471a-a3e9-eb0ae9d550d1 | 21.0.0.21        | 10.35.180.29        | 3c2976af-038d-48c5-868c-3b03c2d08420 |
| 24cba2a0-be1e-4bed-8fc7-8ae51c26b456 | 21.0.0.41        | 10.35.180.49        | f4830dc0-d720-4490-8e8f-02a428089375 |
| 2eaa4233-9c0a-4c5e-9470-3365e8564d64 | 21.0.0.20        | 10.35.180.28        | 7c64b91b-49b0-4879-b8a8-eaaf61cfba40 |
| 330d8c31-6964-4461-9fc3-48afadd3784a | 21.0.0.50        | 10.35.180.58        | 91439e83-23f1-4261-b85b-208d2eb45d25 |
| 37874c65-47f8-4d0a-b58e-5879227966b7 | 21.0.0.13        | 10.35.180.21        | 65e8c540-6482-4e74-9940-26ce8c12f5c0 |
| 392c07c4-b2d7-4d61-af66-dfa5e710df06 | 21.0.0.35        | 10.35.180.43        | d401bb83-3b34-4132-a32a-7f8b4170c359 |
| 3c5fec78-eece-4700-bd23-5a1fb50fe805 | 21.0.0.43        | 10.35.180.51        | 96dbbab7-6aef-4b06-b25e-fcb04df862b8 |
| 40ef46d9-cd67-4329-baff-49aae6d89404 | 21.0.0.29        | 10.35.180.37        | 79bf737b-c30d-41bc-b47a-249b02969a4a |
| 416a3d16-5e9a-4b30-96ff-05491dd58c85 | 21.0.0.32        | 10.35.180.40        | 574955cc-9318-42a9-8f83-3aebf1ae945f |
| 42112145-1a7a-4db8-9d8a-4708778e9cdb | 21.0.0.15        | 10.35.180.23        | 2e1052c3-5aa6-4f89-9aa9-21e91c6cc183 |
| 4393e781-1188-4909-9c33-a311ce3822a0 | 21.0.0.22        | 10.35.180.30        | 475ae6cd-0874-4f71-adc0-42dc1d42e682 |
| 46e18840-3f32-4698-9aca-d1e1b0ccf11d | 21.0.0.39        | 10.35.180.47        | 868ee360-fe34-447e-831f-22e84fd1ffef |
| 4898da79-86ee-426a-8e15-cfec0ab57244 | 21.0.0.48        | 10.35.180.56        | 1e4c4833-1eb2-490a-be59-c814f0eab1e3 |
| 4fa236d8-9c0a-4be8-81ed-d01438ebbccf | 21.0.0.36        | 10.35.180.44        | 3d5b3358-4668-4843-a7dc-045c3b035a4e |
| 52dda9c3-cf58-487d-adfd-9fef783ee748 | 21.0.0.14        | 10.35.180.22        | 836934fa-2b7f-41ea-9706-0f52937af3bd |
| 74c34b60-373e-4193-8832-27054df754a2 | 21.0.0.25        | 10.35.180.33        | 88cba2cd-ae4b-4b97-839f-67f90c792126 |
| 7b101df9-1a6a-48bc-a00f-088af23b7bc1 | 21.0.0.33        | 10.35.180.41        | f5b935a4-95c3-41ee-8135-673150b70723 |
| 8757376c-71d1-488e-91bd-fbfdfde165a6 | 21.0.0.28        | 10.35.180.36        | 85c1a88b-4644-4937-983d-a7e2274843e8 |
| 908e5960-ab20-4bb1-a60d-44cf3468b3ed | 21.0.0.26        | 10.35.180.34        | c309f8b7-b127-4e2d-b82f-665f293880c5 |
| 96c7e216-5ba5-43a9-82c4-9bf19987175a | 21.0.0.30        | 10.35.180.38        | efd9973f-6e6c-433d-8ae9-d10167e486a2 |
| b2bc6fa1-b533-42a3-9bcd-5fb09d0f6fef | 21.0.0.38        | 10.35.180.46        | 81bdbbc8-2caa-41f6-94a0-7d4549dc4deb |
| b37c41e6-83a6-4ce1-acc7-51dd5f16d48a | 21.0.0.51        | 10.35.180.59        | cf66c893-accb-47e0-891d-e46f5340e93d |
| bd23821d-c4e9-4870-985d-f376b6551224 | 21.0.0.52        | 10.35.180.60        | ac885fd5-e316-4e00-bc21-23946085aa49 |
| c2651a2e-c814-442d-a92f-655a78b926a5 | 21.0.0.45        | 10.35.180.53        | 1cf8019b-0fcd-4771-ac43-aa1817e0e24d |
| c65d424f-8bd5-49eb-8612-9e9f8994327a | 21.0.0.23        | 10.35.180.31        | 9b90c2bb-8673-4a2e-810a-3ff5e0698cbd |
| cd27d4bc-67a1-44ab-8053-199f2df9b0d1 | 21.0.0.44        | 10.35.180.52        | e5e9a4ab-6e62-4e0c-98a9-eb24cf78f64a |
| d248c3e9-4baa-49b6-a41b-0a9824e075d3 | 21.0.0.31        | 10.35.180.39        | f7f0ec45-37b7-4d74-bf74-77f9262d41e0 |
| d3428675-76e5-4811-bfa4-33d0c2e6ebf8 | 21.0.0.49        | 10.35.180.57        | c172db6e-37c2-41d3-b966-5196d472f1e4 |
| d5a7ce2b-ca36-403c-a81e-973479b881cb | 21.0.0.37        | 10.35.180.45        | 91b2f6f5-ac3f-41cc-b9f2-69039070554b |
| e01c917f-338c-48cc-a6be-2a9e63a926d8 | 21.0.0.40        | 10.35.180.48        | da61b771-38fd-4684-ae5c-92831c3a2947 |
| e1020508-d2f6-4afc-91e8-45e4541ecd33 | 21.0.0.34        | 10.35.180.42        | d4012e56-e69d-44cc-986a-3cc145e9134b |
| e1e27c62-8163-4f55-aadd-ccf757aa86ce | 21.0.0.47        | 10.35.180.55        | 85db3d86-363b-41b3-8c7d-54f505a6429f |
| feedf053-1edb-4f23-a136-18641d236738 | 21.0.0.46        | 10.35.180.54        | f7d6ab92-6ed5-4a08-a110-428d48fc0a93 |
+--------------------------------------+------------------+---------------------+--------------------------------------+
[root@puma04 ~(keystone_admin_tenant1)]$for i in `seq 40`; do neutron port-delete port$i; done
Deleted port: port1
Deleted port: port2
Deleted port: port3
Deleted port: port4
Deleted port: port5
Deleted port: port6
Deleted port: port7
Deleted port: port8
Deleted port: port9
Deleted port: port10
Deleted port: port11
Deleted port: port12
Deleted port: port13
Deleted port: port14
Deleted port: port15
Deleted port: port16
Deleted port: port17
Deleted port: port18
Deleted port: port19
Deleted port: port20
Deleted port: port21
Deleted port: port22
Deleted port: port23
Deleted port: port24
Deleted port: port25
Deleted port: port26
Deleted port: port27
Deleted port: port28
Deleted port: port29
Deleted port: port30
Deleted port: port31
Deleted port: port32
Deleted port: port33
Deleted port: port34
Deleted port: port35
Deleted port: port36
Deleted port: port37
Deleted port: port38
Deleted port: port39
Deleted port: port40
[root@puma04 ~(keystone_admin_tenant1)]$neutron floatingip-list
+--------------------------------------+------------------+---------------------+---------+
| id                                   | fixed_ip_address | floating_ip_address | port_id |
+--------------------------------------+------------------+---------------------+---------+
| 0a4462a2-aafe-4bbd-a0cb-a88a1222b112 |                  | 10.35.180.26        |         |
| 0c7eedde-bd3e-4df4-89c1-d40f640999e8 |                  | 10.35.180.50        |         |
| 176515be-ae2e-4f78-ae13-04d1502d1431 |                  | 10.35.180.35        |         |
| 19adb669-0d85-46f8-8535-e0b9b02e499c |                  | 10.35.180.25        |         |
| 1c0c1999-8c53-46d3-80ec-e0cd95a3af88 |                  | 10.35.180.27        |         |
| 1c5590f6-5745-41a8-b2c9-8e7d085cf997 |                  | 10.35.180.32        |         |
| 1d88c18e-87de-4815-b984-e09584551b7b |                  | 10.35.180.24        |         |
| 20cdfe3b-5616-471a-a3e9-eb0ae9d550d1 |                  | 10.35.180.29        |         |
| 24cba2a0-be1e-4bed-8fc7-8ae51c26b456 |                  | 10.35.180.49        |         |
| 2eaa4233-9c0a-4c5e-9470-3365e8564d64 |                  | 10.35.180.28        |         |
| 330d8c31-6964-4461-9fc3-48afadd3784a |                  | 10.35.180.58        |         |
| 37874c65-47f8-4d0a-b58e-5879227966b7 |                  | 10.35.180.21        |         |
| 392c07c4-b2d7-4d61-af66-dfa5e710df06 |                  | 10.35.180.43        |         |
| 3c5fec78-eece-4700-bd23-5a1fb50fe805 |                  | 10.35.180.51        |         |
| 40ef46d9-cd67-4329-baff-49aae6d89404 |                  | 10.35.180.37        |         |
| 416a3d16-5e9a-4b30-96ff-05491dd58c85 |                  | 10.35.180.40        |         |
| 42112145-1a7a-4db8-9d8a-4708778e9cdb |                  | 10.35.180.23        |         |
| 4393e781-1188-4909-9c33-a311ce3822a0 |                  | 10.35.180.30        |         |
| 46e18840-3f32-4698-9aca-d1e1b0ccf11d |                  | 10.35.180.47        |         |
| 4898da79-86ee-426a-8e15-cfec0ab57244 |                  | 10.35.180.56        |         |
| 4fa236d8-9c0a-4be8-81ed-d01438ebbccf |                  | 10.35.180.44        |         |
| 52dda9c3-cf58-487d-adfd-9fef783ee748 |                  | 10.35.180.22        |         |
| 74c34b60-373e-4193-8832-27054df754a2 |                  | 10.35.180.33        |         |
| 7b101df9-1a6a-48bc-a00f-088af23b7bc1 |                  | 10.35.180.41        |         |
| 8757376c-71d1-488e-91bd-fbfdfde165a6 |                  | 10.35.180.36        |         |
| 908e5960-ab20-4bb1-a60d-44cf3468b3ed |                  | 10.35.180.34        |         |
| 96c7e216-5ba5-43a9-82c4-9bf19987175a |                  | 10.35.180.38        |         |
| b2bc6fa1-b533-42a3-9bcd-5fb09d0f6fef |                  | 10.35.180.46        |         |
| b37c41e6-83a6-4ce1-acc7-51dd5f16d48a |                  | 10.35.180.59        |         |
| bd23821d-c4e9-4870-985d-f376b6551224 |                  | 10.35.180.60        |         |
| c2651a2e-c814-442d-a92f-655a78b926a5 |                  | 10.35.180.53        |         |
| c65d424f-8bd5-49eb-8612-9e9f8994327a |                  | 10.35.180.31        |         |
| cd27d4bc-67a1-44ab-8053-199f2df9b0d1 |                  | 10.35.180.52        |         |
| d248c3e9-4baa-49b6-a41b-0a9824e075d3 |                  | 10.35.180.39        |         |
| d3428675-76e5-4811-bfa4-33d0c2e6ebf8 |                  | 10.35.180.57        |         |
| d5a7ce2b-ca36-403c-a81e-973479b881cb |                  | 10.35.180.45        |         |
| e01c917f-338c-48cc-a6be-2a9e63a926d8 |                  | 10.35.180.48        |         |
| e1020508-d2f6-4afc-91e8-45e4541ecd33 |                  | 10.35.180.42        |         |
| e1e27c62-8163-4f55-aadd-ccf757aa86ce |                  | 10.35.180.55        |         |
| feedf053-1edb-4f23-a136-18641d236738 |                  | 10.35.180.54        |         |
+--------------------------------------+------------------+---------------------+---------+

Comment 10 errata-xmlrpc 2014-07-28 18:58:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0953.html

Note You need to log in before you can comment on or make changes to this bug.