Bug 1327395

beaker-sync-tasks crashes after syncing a few hundred tasks, like this:

2016-04-15 04:08:43,110 - Error syncing tasks. Got error Failed to import,  1.0-1 is the same version we already have
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bkr/server/tools/sync_tasks.py", line 199, in sync_tasks
    self.tasklib.update_tasks(tasks_and_writes_current_batch)
  File "/usr/lib/python2.6/site-packages/bkr/server/model/tasklibrary.py", line 224, in update_tasks
    task, downgrade = Task.create_from_taskinfo(self.read_taskinfo(f))
  File "/usr/lib/python2.6/site-packages/bkr/server/model/tasklibrary.py", line 415, in create_from_taskinfo
    raise BX(_("Failed to import,  %s is the same version we already have" % task.version))
BX: Failed to import,  1.0-1 is the same version we already have

Even better, it does that after spending several hours(!) loading thousands of task XMLs into memory from the remote Beaker instance, one per XMLRPC call.

If the remote version and local version are the same it should just be skipping the task.

I hacked around that first problem and ran it again, and now it has crashed in another interesting and possibly related way:

2016-04-15 04:30:56,290 - (IntegrityError) (1062, "Duplicate entry '17142-33' for key 'PRIMARY'") 'INSERT INTO task_type_map (task_id, task_type_id) VALUES (%s, %s)' ((17186L, 35L), (17178L, 621L), (17172L, 1L), (17167L, 2L), (9974L, 33L), (6889L, 33L), (17138L, 33L), (17142L, 33L)  ... displaying 10 of 70 total bound parameter sets ...  (17139L, 33L), (17169L, 2L))
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bkr/server/tools/sync_tasks.py", line 238, in tasks_add
    self.sync_tasks(tasks_to_sync)
  File "/usr/lib/python2.6/site-packages/bkr/server/tools/sync_tasks.py", line 205, in sync_tasks
    session.commit()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/scoping.py", line 149, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 721, in commit
    self.transaction.commit()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in commit
    self._prepare_impl()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl
    self.session.flush()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1824, in flush
    self._flush(objects)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1942, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1906, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
    rec.execute(self)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 481, in execute
    self.dependency_processor.process_saves(uow, states)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/dependency.py", line 1083, in process_saves
    secondary_update, secondary_delete)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/dependency.py", line 1126, in _run_crud
    connection.execute(statement, secondary_insert)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute
    params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 856, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/connectors/mysqldb.py", line 60, in do_executemany
    rowcount = cursor.executemany(statement, parameters)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 205, in executemany
    r = r + self.execute(query, a)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (IntegrityError) (1062, "Duplicate entry '17142-33' for key 'PRIMARY'") 'INSERT INTO task_type_map (task_id, task_type_id) VALUES (%s, %s)' ((17186L, 35L), (17178L, 621L), (17172L, 1L), (17167L, 2L), (9974L, 33L), (6889L, 33L), (17138L, 33L), (17142L, 33L)  ... displaying 10 of 70 total bound parameter sets ...  (17139L, 33L), (17169L, 2L))

This defeated my first hack to keep it running in spite of problematic tasks (since it occurs during flush which is done by commit).

The whole thing really needs some love, because right now it's totally unusable.