1296708 – Rapid creation/deletion of routers in Neutron fails

Bug 1296708 - Rapid creation/deletion of routers in Neutron fails

Summary: Rapid creation/deletion of routers in Neutron fails

Keywords:
Status:	CLOSED DUPLICATE of bug 1281254
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	async
Target Release:	7.0 (Kilo)
Assignee:	John Schwarz
QA Contact:	Ofer Blaut
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-01-07 23:37 UTC by Graeme Gillies
Modified:	2016-06-04 02:49 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-06-04 02:49:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1523780	0	None	None	None	2016-01-07 23:37:17 UTC

Description Graeme Gillies 2016-01-07 23:37:17 UTC

The following is an explanation of the issue from the cloud operator

====
I'm able to reproduce the neutron router creation issue that Yair is having.  Yair pointed to:

https://github.com/ansible/ansible-modules-core/blob/devel/cloud/openstack/_quantum_router.py#L163-L173

On which I have based the following modified python:

> #!/usr/bin/python
> #coding: utf-8 -*-
> 
> try:
>     try:
>         from neutronclient.neutron import client
>     except ImportError:
>         from quantumclient.quantum import client
>     from keystoneclient.v2_0 import client as ksclient
>     HAVE_DEPS = True
> except ImportError:
>     HAVE_DEPS = False
> 
> _os_keystone = None
> _os_tenant_id = None
> 
> def _get_ksclient():
>     try:
>         kclient = ksclient.Client(username='admin',
>                                  password='CHANGE',
>                                  tenant_name='admin',
>                                  auth_url='http://10.8.188.11:5000/v2.0/')
>     except Exception, e:
>         print "Error authenticating to the keystone: %s " % e.message
>     global _os_keystone
>     _os_keystone = kclient
>     return kclient
> 
> 
> def _get_endpoint(ksclient):
>     try:
>         endpoint = ksclient.service_catalog.url_for(service_type='network', endpoint_type='publicURL')
>     except Exception, e:
>         print "Error getting network endpoint: %s" % e.message
>     return endpoint
> 
> def _get_neutron_client():
>     _ksclient = _get_ksclient()
>     token = _ksclient.auth_token
>     endpoint = _get_endpoint(_ksclient)
>     kwargs = {
>             'token': token,
>             'endpoint_url': endpoint
>     }
>     try:
>         neutron = client.Client('2.0', **kwargs)
>     except Exception, e:
>         print "Error in connecting to neutron: %s " % e.message
>     return neutron
> 
> def _create_router(neutron):
>     router = {
>             'name': "somerouter1",
>             'tenant_id': 'admin',
>             'admin_state_up': True,
>     }
>     try:
>         new_router = neutron.create_router(dict(router=router))
>     except Exception, e:
>         print "something went wrong."
>     return new_router['router']['id']
> 
> def _delete_router(neutron, router_id):
>     try:
>         neutron.delete_router(router_id)
>     except:
>         print "Error in deleting the router"
>     return True
> 
> def main():
>     neutron = _get_neutron_client()
>     router_id = _create_router(neutron)
>     _delete_router(neutron, router_id)
> if __name__ == '__main__':
>     main()

The above python code creates and deletes a router (see the _create_router() function).  To reproduce the issue, I created 5 copies of the above script, updating the name of the router to a unique name in each script (the code can be updated to generate a random string for the router name), and used the following to run 5 concurrent router creation loops:

main script:

> #!/bin/sh
> 
> for i in 1 2 3 4 5 ; do 
> 
> ./run-loop.sh router$i.py & 
> 
> done

run-loop.sh script:

> #!/bin/sh
> 
> while true ; do
> 
> python $1
> sleep 2
> 
> done
===

Further investigation reveals this is highly likely to be this upstream issue

https://bugs.launchpad.net/neutron/+bug/1523780

So we would love to get this backported to 7.2

Comment 3 Graeme Gillies 2016-01-08 05:43:24 UTC

The following output was taken from the database of the environment, showing that according to mysql, this might actually be a foreign key error not a deadlock, though someone more knowledgeable will need to confirm

=====================================
160108  0:38:20 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 35 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 96988 1_second, 96987 sleeps, 9693 10_second, 60 background, 60 flush
srv_master_thread log flush and writes: 95002
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 16401, signal count 19864
Mutex spin waits 251598, rounds 882115, OS waits 2574
RW-shared spins 17511, rounds 396158, OS waits 11765
RW-excl spins 3386, rounds 86448, OS waits 1676
Spin rounds per wait: 3.51 mutex, 22.62 RW-shared, 25.53 RW-excl
------------------------
LATEST FOREIGN KEY ERROR
------------------------
160107 14:12:36 Transaction:
TRANSACTION 3F45C1, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1248, 2 row lock(s), undo log entries 1
MySQL thread id 123452, OS thread handle 0x7fe487565700, query id 13771080 172.16.104.16 neutron update
INSERT INTO ha_router_agent_port_bindings (port_id, router_id, l3_agent_id, state) VALUES ('7f923923-3d18-495d-bee7-9c2116084cf3', 'dd781745-a569-4cca-9ca2-9ad92efdcda0', NULL, 'standby')
Foreign key constraint fails for table `ovs_neutron`.`ha_router_agent_port_bindings`:
,
  CONSTRAINT `ha_router_agent_port_bindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE
Trying to add in child table, in index `router_id` tuple:
DATA TUPLE: 2 fields;
 0: len 36; hex 64643738313734352d613536392d346363612d396361322d396164393265666463646130; asc dd781745-a569-4cca-9ca2-9ad92efdcda0;;
 1: len 36; hex 37663932333932332d336431382d343935642d626565372d396332313136303834636633; asc 7f923923-3d18-495d-bee7-9c2116084cf3;;

But in parent table `ovs_neutron`.`routers`, in index `PRIMARY`,
the closest match we can find is record:
PHYSICAL RECORD: n_fields 9; compact format; info bits 0
 0: len 30; hex 37393061613263632d386432372d343931622d613563362d333561303335; asc 790aa2cc-8d27-491b-a5c6-35a035; (total 36 bytes);
 1: len 6; hex 0000003f44a9; asc    ?D ;;
 2: len 7; hex e3000002600110; asc     `  ;;
 3: len 5; hex 61646d696e; asc admin;;
 4: len 11; hex 736f6d65726f7574657232; asc somerouter2;;
 5: len 6; hex 414354495645; asc ACTIVE;;
 6: len 1; hex 81; asc  ;;
 7: SQL NULL;
 8: len 1; hex 81; asc  ;;

Comment 4 Graeme Gillies 2016-01-08 06:03:45 UTC

Further indication of a foreign key error.

from the neutron l3 agent logs

2016-01-07 14:01:28.464 143913 ERROR oslo_messaging.rpc.dispatcher [req-1a1ccc00-fae6-4c03-a75f-eb36c9e48f8d ] Exception during message handling: (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ha_router_agent_port_bindings`, CONSTRAINT `ha_router_agent_port_bindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') [SQL: u'INSERT INTO ha_router_agent_port_bindings (port_id, router_id, l3_agent_id, state) VALUES (%s, %s, %s, %s)'] [parameters: ('94e4a3fd-8942-4d6f-b66f-13d5df28066a', '3a2233f1-5d8a-4ff5-bac9-70712e13e1d0', None, 'standby')]
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 78, in sync_routers
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self.l3plugin.auto_schedule_routers(context, host, router_ids)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 487, in auto_schedule_routers
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self, context, host, router_ids)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 160, in auto_schedule_routers
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self.bind_routers(context, plugin, target_routers, l3_agent)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 205, in bind_routers
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     router['tenant_id'], l3_agent)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 297, in create_ha_router_binding
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     ha_network.network.id, tenant_id)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/l3_hamode_db.py", line 331, in add_ha_port
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     l3_port_check=False)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/l3_hamode_db.py", line 327, in add_ha_port
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     return self._create_ha_port_binding(context, port['id'], router_id)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/l3_hamode_db.py", line 310, in _create_ha_port_binding
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     context.session.add(portbinding)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self.rollback()
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self.commit()
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self._prepare_impl()
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self.session.flush()
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self._flush(objects)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     transaction.rollback(_capture_exception=True)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     flush_context.execute()
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     rec.execute(self)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     uow
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     mapper, table, insert)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748, in _emit_insert_statements
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     execute(statement, multiparams)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     return meth(self, multiparams, params)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     return connection._execute_clauseelement(self, multiparams, params)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     compiled_sql, distilled_params
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     context)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     util.raise_from_cause(newraise, exc_info)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     reraise(type(exception), exception, tb=exc_tb)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     context)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     cursor.execute(statement, parameters)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     self.errorhandler(self, exc, value)
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher     raise errorclass, errorvalue
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher DBReferenceError: (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ha_router_agent_port_bindings`, CONSTRAINT `ha_router_agent_port_bindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') [SQL: u'INSERT INTO ha_router_agent_port_bindings (port_id, router_id, l3_agent_id, state) VALUES (%s, %s, %s, %s)'] [parameters: ('94e4a3fd-8942-4d6f-b66f-13d5df28066a', '3a2233f1-5d8a-4ff5-bac9-70712e13e1d0', None, 'standby')]
2016-01-07 14:01:28.464 143913 TRACE oslo_messaging.rpc.dispatcher
2016-01-07 14:01:28.492 143913 ERROR oslo_messaging._drivers.common [req-1a1ccc00-fae6-4c03-a75f-eb36c9e48f8d ] Returning exception (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ha_router_agent_port_bindings`, CONSTRAINT `ha_router_agent_port_bindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') [SQL: u'INSERT INTO ha_router_agent_port_bindings (port_id, router_id, l3_agent_id, state) VALUES (%s, %s, %s, %s)'] [parameters: ('94e4a3fd-8942-4d6f-b66f-13d5df28066a', '3a2233f1-5d8a-4ff5-bac9-70712e13e1d0', None, 'standby')] to caller
2016-01-07 14:01:28.492 143913 ERROR oslo_messaging._drivers.common [req-1a1ccc00-fae6-4c03-a75f-eb36c9e48f8d ] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 78, in sync_routers\n    self.l3plugin.auto_schedule_routers(context, host, router_ids)\n', '  File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 487, in auto_schedule_routers\n    self, context, host, router_ids)\n', '  File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 160, in auto_schedule_routers\n    self.bind_routers(context, plugin, target_routers, l3_agent)\n', '  File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 205, in bind_routers\n    router[\'tenant_id\'], l3_agent)\n', '  File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 297, in create_ha_router_binding\n    ha_network.network.id, tenant_id)\n', '  File "/usr/lib/python2.7/site-packages/neutron/db/l3_hamode_db.py", line 331, in add_ha_port\n    l3_port_check=False)\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/neutron/db/l3_hamode_db.py", line 327, in add_ha_port\n    return self._create_ha_port_binding(context, port[\'id\'], router_id)\n', '  File "/usr/lib/python2.7/site-packages/neutron/db/l3_hamode_db.py", line 310, in _create_ha_port_binding\n    context.session.add(portbinding)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__\n    self.rollback()\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__\n    self.commit()\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit\n    self._prepare_impl()\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl\n    self.session.flush()\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush\n    self._flush(objects)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush\n    transaction.rollback(_capture_exception=True)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush\n    flush_context.execute()\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute\n    rec.execute(self)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute\n    uow\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj\n    mapper, table, insert)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748, in _emit_insert_statements\n    execute(statement, multiparams)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute\n    return meth(self, multiparams, params)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n    return connection._execute_clauseelement(self, multiparams, params)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement\n    compiled_sql, distilled_params\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context\n    context)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception\n    util.raise_from_cause(newraise, exc_info)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context\n    context)\n', '  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute\n    cursor.execute(statement, parameters)\n', '  File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute\n    self.errorhandler(self, exc, value)\n', '  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n    raise errorclass, errorvalue\n', "DBReferenceError: (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ovs_neutron`.`ha_router_agent_port_bindings`, CONSTRAINT `ha_router_agent_port_bindings_ibfk_2` FOREIGN KEY (`router_id`) REFERENCES `routers` (`id`) ON DELETE CASCADE)') [SQL: u'INSERT INTO ha_router_agent_port_bindings (port_id, router_id, l3_agent_id, state) VALUES (%s, %s, %s, %s)'] [parameters: ('94e4a3fd-8942-4d6f-b66f-13d5df28066a', '3a2233f1-5d8a-4ff5-bac9-70712e13e1d0', None, 'standby')]\n"]

Comment 5 Assaf Muller 2016-06-04 02:49:30 UTC


*** This bug has been marked as a duplicate of bug 1281254 ***

Note You need to log in before you can comment on or make changes to this bug.