1373258 – Race condition in L3HA while creating and deleting routers.

Bug 1373258 - Race condition in L3HA while creating and deleting routers.

Summary: Race condition in L3HA while creating and deleting routers.

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	7.0 (Kilo)
Assignee:	Assaf Muller
QA Contact:	Toni Freger
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-09-05 15:39 UTC by John Schwarz
Modified:	2017-04-26 21:41 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-04-26 21:41:03 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description John Schwarz 2016-09-05 15:39:27 UTC

While creating and deleting an HA router, a StaleDataError can occur.

This happens starting openstack-neutron-2015.1.4-3.el7ost (following the fix for bz 1281254).

The trace is:
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 78, in sync_routers
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self.l3plugin.auto_schedule_routers(context, host, router_ids)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 511, in auto_schedule_routers
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self, context, host, router_ids)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 157, in auto_schedule_routers
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     plugin, context, l3_agent)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 343, in schedule_ha_routers_to_additional_agent
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     agent)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 321, in create_ha_router_binding
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     port_binding.l3_agent_id = agent['id']
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self.rollback()
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self.commit()
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self._prepare_impl()
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self.session.flush()
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     self._flush(objects)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     transaction.rollback(_capture_exception=True)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     compat.reraise(exc_type, exc_value, exc_tb)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     flush_context.execute()
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     rec.execute(self)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     uow
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     mapper, table, update)
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 712, in _emit_update_statements
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher     (table.description, len(records), rows))
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher StaleDataError: UPDATE statement on table 'ha_router_agent_port_bindings' expected to update 1 row(s); 0 were matched.
2016-09-05 06:52:35.646 22198 TRACE oslo_messaging.rpc.dispatcher 
2016-09-05 06:52:35.647 22198 ERROR oslo_messaging._drivers.common [req-4c2a51b3-df0b-4638-8ecd-facba58686d2 ] Returning exception UPDATE statement on table 'ha_router_agent_port_bindings' expected to update 1 row(s); 0 were matched. to caller

Comment 2 Assaf Muller 2017-01-13 19:08:40 UTC

Status update?

Comment 3 John Schwarz 2017-01-15 09:33:35 UTC

The patch [1] was supposed to solve it, but the backport proved to be non-trivial (Jakub had some comments against the current implementation that required quite a bit of work). Do we still want to fix this?

[1]: https://code.engineering.redhat.com/gerrit/#/c/83440/

Comment 4 Assaf Muller 2017-01-15 18:55:21 UTC

(In reply to John Schwarz from comment #3)
> The patch [1] was supposed to solve it, but the backport proved to be
> non-trivial (Jakub had some comments against the current implementation that
> required quite a bit of work). Do we still want to fix this?
> 
> [1]: https://code.engineering.redhat.com/gerrit/#/c/83440/

Yes, we do.

Comment 5 Assaf Muller 2017-04-26 21:41:03 UTC

Fixed in later OSP versions, the backport proved to be too complicated.

Note You need to log in before you can comment on or make changes to this bug.