Description of problem: On running perf/scale tests on an OSP 12 cloud with 3 OpenStack controllers + 3 ODLs + 28 computes, by creating 100s of neutron resources and deleting them we see several StaleDataErrors in the neutron-server logs. Version-Release number of selected component (if applicable): OSP12 Puddle: 2017-10-31.2 ODL RPM: opendaylight-6.2.0-3.el7ost.noarch python-networking-odl-11.0.1-0.20171017155941.el7ost.noarch How reproducible: Steps to Reproduce: 1. Run perf/scale tests using Browbeat+neutron and create large number of neutro nresources 2. 3. Actual results: Errors in logs Expected results: No errors Additional info: 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal [req-0ed305a2-9f63-4b3a-bec6-370a271fde5a - - - - -] Error on run_sync_thread: StaleDataError: UPDATE statement on table 'opendaylightjournal' expected to update 1 row(s); 0 were matched. 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal Traceback (most recent call last): 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/journal/journal.py", line 261, in run_sync_thread 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self.sync_pending_entries() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/journal/journal.py", line 279, in sync_pending_entries 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal entry = db.get_oldest_pending_db_row_with_lock(context.session) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal setattr(e, '_RETRY_EXCEEDED', True) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self.force_reraise() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal return f(*args, **kwargs) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal ectxt.value = e.inner_exc 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self.force_reraise() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal return f(*args, **kwargs) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self.force_reraise() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal return f(*dup_args, **dup_kwargs) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/db/db.py", line 86, in get_oldest_pending_db_row_with_lock 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal update_db_row_state(session, row, odl_const.PROCESSING) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/db/db.py", line 102, in update_db_row_state 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal session.merge(row) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1839, in merge 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self._autoflush() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1397, in _autoflush 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self.flush() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in flush 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal self._flush(objects) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal transaction.rollback(_capture_exception=True) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal compat.reraise(exc_type, exc_value, exc_tb) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal flush_context.execute() 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal rec.execute(self) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal uow 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal mapper, table, update) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal (table.description, len(records), rows)) 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal StaleDataError: UPDATE statement on table 'opendaylightjournal' expected to update 1 row(s); 0 were matched. 2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal 2017-11-10 19:16:02.277 721543 DEBUG networking_odl.journal.journal [req-0ed305a2-9f63-4b3a-bec6-370a271fde5a - - - - -] Start processing journal entries sync_pending_entries /usr/lib/python2.7/site-packages/networking_odl/journal/journal.py:267
*** This bug has been marked as a duplicate of bug 1487783 ***