Bug 1512085 - Error on run_sync_thread: StaleDataError
Summary: Error on run_sync_thread: StaleDataError
Keywords:
Status: CLOSED DUPLICATE of bug 1487783
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-odl
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Mike Kolesnik
QA Contact: Itzik Brown
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 19:43 UTC by Sai Sindhur Malleni
Modified: 2018-10-24 12:41 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
N/A
Last Closed: 2018-02-19 11:36:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sai Sindhur Malleni 2017-11-10 19:43:14 UTC
Description of problem:
On running perf/scale tests on an OSP 12 cloud with 3 OpenStack controllers + 3 ODLs + 28 computes, by creating 100s of neutron resources and deleting them we see several StaleDataErrors in the neutron-server logs.  

Version-Release number of selected component (if applicable):
OSP12
Puddle: 2017-10-31.2
ODL RPM: opendaylight-6.2.0-3.el7ost.noarch
python-networking-odl-11.0.1-0.20171017155941.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. Run perf/scale tests using Browbeat+neutron and create large number of neutro nresources
2.
3.

Actual results:
Errors in logs

Expected results:
No errors

Additional info:
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal [req-0ed305a2-9f63-4b3a-bec6-370a271fde5a - - - - -] Error on run_sync_thread: StaleDataError: UPDATE statement on table 'opendaylightjournal' expected to update 1 row(s); 0 were matched.
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal Traceback (most recent call last):
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/journal/journal.py", line 261, in run_sync_thread
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self.sync_pending_entries()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/journal/journal.py", line 279, in sync_pending_entries
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     entry = db.get_oldest_pending_db_row_with_lock(context.session)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 93, in wrapped
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     setattr(e, '_RETRY_EXCEEDED', True)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self.force_reraise()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     six.reraise(self.type_, self.value, self.tb)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 89, in wrapped
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     return f(*args, **kwargs)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     ectxt.value = e.inner_exc
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self.force_reraise()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     six.reraise(self.type_, self.value, self.tb)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     return f(*args, **kwargs)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 128, in wrapped
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self.force_reraise()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     six.reraise(self.type_, self.value, self.tb)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 124, in wrapped
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     return f(*dup_args, **dup_kwargs)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/db/db.py", line 86, in get_oldest_pending_db_row_with_lock
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     update_db_row_state(session, row, odl_const.PROCESSING)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/db/db.py", line 102, in update_db_row_state
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     session.merge(row)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1839, in merge
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self._autoflush()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1397, in _autoflush
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self.flush()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2171, in flush
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     self._flush(objects)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2291, in _flush
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     transaction.rollback(_capture_exception=True)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     compat.reraise(exc_type, exc_value, exc_tb)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2255, in _flush
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     flush_context.execute()
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     rec.execute(self)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     uow
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     mapper, table, update)
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal     (table.description, len(records), rows))
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal StaleDataError: UPDATE statement on table 'opendaylightjournal' expected to update 1 row(s); 0 were matched.
2017-11-10 19:16:02.276 721543 ERROR networking_odl.journal.journal
2017-11-10 19:16:02.277 721543 DEBUG networking_odl.journal.journal [req-0ed305a2-9f63-4b3a-bec6-370a271fde5a - - - - -] Start processing journal entries sync_pending_entries /usr/lib/python2.7/site-packages/networking_odl/journal/journal.py:267

Comment 1 Mike Kolesnik 2018-02-19 11:36:46 UTC

*** This bug has been marked as a duplicate of bug 1487783 ***


Note You need to log in before you can comment on or make changes to this bug.