Description of problem:

If retrace_log is not created, the cleanup job crashes and does not run properly.  This affects cleaning up all other jobs since the cleanup task crashes if any task directory does not contain retrace_log.

Traceback (most recent call last):
  File "/usr/bin/retrace-server-cleanup", line 71, in <module>
    task.set_log("Killed by garbage collector\n", True)
  File "/usr/lib/python2.6/site-packages/retrace/retrace.py", line 1431, in set_log
    self.set_atomic(RetraceTask.LOG_FILE, log, mode=mode)
  File "/usr/lib/python2.6/site-packages/retrace/retrace.py", line 1334, in set_atomic
    shutil.copyfile(filename, tmpfilename)
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/cores/retrace/tasks/396908399/retrace_log'
Traceback (most recent call last):
  File "/usr/bin/retrace-server-cleanup", line 71, in <module>
    task.set_log("Killed by garbage collector\n", True)
  File "/usr/lib/python2.6/site-packages/retrace/retrace.py", line 1431, in set_log
    self.set_atomic(RetraceTask.LOG_FILE, log, mode=mode)
  File "/usr/lib/python2.6/site-packages/retrace/retrace.py", line 1334, in set_atomic
    shutil.copyfile(filename, tmpfilename)
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/cores/retrace/tasks/396908399/retrace_log'
Traceback (most recent call last):
/var/log/retrace-server/cleanup_error.log


Version-Release number of selected component (if applicable):
retrace-server-1.10-1.el6.noarch


How reproducible:
Every time if any directory does not contain retrace_log


Steps to Reproduce:
1. Some task does not complete properly such that retrace_log is not created
2.
3.

Actual results:
retrace cleanup job stops running and no tasks are cleaned up after encountering the task which does not have retrace_log

Expected results:
cleanup job should continue to run even if some directory does not contain this file

Additional info:
I opened a ticket for this and the admins manually moved / removed the tasks missing retrace_log, but we need a longer term fix as this may happen again.