Bug 1086321

Assuming Michal you're ok with the design of a second config variable, I think we just need a similar section in retrace-server-cleanup to the existing one for DeleteTaskAfter, something like the below.

To be conservative we could just go with STATUS_FAILED but I'm not sure.  If we got with the 'non-success' idea, we may need to check for a minimum value due to longer running tasks which may still be in progress.  We would not want this addition to the cleanup job cleaning up tasks which may still be in progress.

diff -Nurp /usr/bin/retrace-server-cleanup /usr/bin/retrace-server-cleanup.bz1086321
--- /usr/bin/retrace-server-cleanup     2014-03-25 09:04:07.000000000 -0400
+++ /usr/bin/retrace-server-cleanup.bz1086321   2014-04-17 09:16:56.019129321 -0400
@@ -123,3 +123,21 @@ if __name__ == "__main__":
                 if task.get_age() >= CONFIG["DeleteTaskAfter"]:
                     log.write("Deleting old task %s\n" % filename)
                     task.remove()
+
+        if CONFIG["DeleteFailedTaskAfter"] > 0:
+            # clean up old failed tasks
+            try:
+                files = os.listdir(CONFIG["SaveDir"])
+            except OSError, ex:
+                files = []
+                log.write("Error listing task directory: %s\n" % ex)
+
+            for filename in files:
+                try:
+                    task = RetraceTask(filename)
+                except:
+                    continue
+
+                if task.get_age() >= CONFIG["DeleteNonSuccessTaskAfter"] and task.get_status == STATUS_FAILED:
+                    log.write("Deleting old non-success task %s\n" % filename)
+                    task.remove()

(In reply to Dave Wysochanski from comment #3)
> Assuming Michal you're ok with the design of a second config variable, I
> think we just need a similar section in retrace-server-cleanup to the
> existing one for DeleteTaskAfter, something like the below.
> 
> To be conservative we could just go with STATUS_FAILED but I'm not sure.  If
> we got with the 'non-success' idea, we may need to check for a minimum value
> due to longer running tasks which may still be in progress.  We would not
> want this addition to the cleanup job cleaning up tasks which may still be
> in progress.
> 
> diff -Nurp /usr/bin/retrace-server-cleanup
> /usr/bin/retrace-server-cleanup.bz1086321
> --- /usr/bin/retrace-server-cleanup     2014-03-25 09:04:07.000000000 -0400
> +++ /usr/bin/retrace-server-cleanup.bz1086321   2014-04-17
> 09:16:56.019129321 -0400
> @@ -123,3 +123,21 @@ if __name__ == "__main__":
>                  if task.get_age() >= CONFIG["DeleteTaskAfter"]:
>                      log.write("Deleting old task %s\n" % filename)
>                      task.remove()
> +
> +        if CONFIG["DeleteFailedTaskAfter"] > 0:
> +            # clean up old failed tasks
> +            try:
> +                files = os.listdir(CONFIG["SaveDir"])
> +            except OSError, ex:
> +                files = []
> +                log.write("Error listing task directory: %s\n" % ex)
> +
> +            for filename in files:
> +                try:
> +                    task = RetraceTask(filename)
> +                except:
> +                    continue
> +
> +                if task.get_age() >= CONFIG["DeleteNonSuccessTaskAfter"]
> and task.get_status == STATUS_FAILED:
> +                    log.write("Deleting old non-success task %s\n" %
> filename)
> +                    task.remove()

The above patch does not even work but I will attach a cleaned up / tested patch soon.

Created attachment 887198 [details]
Patch to add a new config variable and section to allow for more aggressive deletes of failed tasks

Test case / outline:
1. On retrace-server system, run script to find failed tasks (see attachment).  The system should have at least one failed task which is newer than the value of DeleteTaskAfter
2. Add DeleteFailedTaskAfter to /etc/retrace-server.conf with a value just below the value of DeleteTaskAfter and such that it should remove the failed task
3. Run the cleanup script, look at the cleanup log, and make sure you see something like the below.  On my system the log is /var/log/retrace-server/cleanup.log
# tail /var/log/retrace-server/cleanup.log
...
[2014-04-17 10:01:13] Running cleanup
Deleting old failed task 217408639

Looks fixed in retrace-server-1.11-4.el6.noarch

retrace-server-1.12-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/retrace-server-1.12-2.el6

retrace-server-1.12-2.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.