Hide Forgot
Description of problem: Today retrace-server has cleanup task which removes 'old' tasks based on the mtime of the task directory. This is good but does not cover the situation where tasks fail. Task that fail are often submitted multiple times, and so the space usage can be significant. Also these failed tasks are not useful at all - they are just dead space. On our system where we have set to delete after 180 days: $ grep ^Delete /etc/retrace-server.conf DeleteTaskAfter = 4320 we see if we would more aggressively remove failed tasks within 30 days, (tasks where 'status' file contains 7), we would retain a savings of around 2TB out of 14 TB. This is significant savings and worth considering a separate cleanup setting. Recommendation is to add a new 'DeleteNonSuccessTaskAfter' setting to remove tasks with 'status != STATUS_SUCCESS == 6' after the timeframe given by the setting. One note of warning. There probably needs to be a minimum value here just in case someone sets this too aggressively low. For example we probably would not want someone to be able to set this value to 1 since it may delete long running tasks just submitted (tasks taking an hour to process). Then again, if a task takes an hour to process, is there another bug that needs filed? Maybe 24 - 48 is a good 'minimum' just in case something goes wrong with the status value and tasks are really usable? In short, I don't think there is any use to keep tasks with status != STATUS_SUCCESS after we're sure we've had enough time to process them. Could there be exceptions to this rule? I'm not sure but maybe not worth worrying about since they imply other bugs. One such exception is if the 'status' value is unreliable - we could end up deleting tasks which are really usable. Version-Release number of selected component (if applicable): retrace-server-1.11-3.el6.noarch How reproducible: Fairly noticeable on our production system. Steps to Reproduce: Search for tasks with 'status != 6'. These tasks are not useable vmcores AFAIK. Actual results: Failed tasks only get deleted with the normal cleanup processing. Expected results: Failed tasks should have more aggressive cleanup. Additional info: I think Brian R. has some scripts which look for failed tasks. We will probably implement some sort of script cleanup in the interim until this gets fixed, unless it is easy and can be addressed shortly.
Assuming Michal you're ok with the design of a second config variable, I think we just need a similar section in retrace-server-cleanup to the existing one for DeleteTaskAfter, something like the below. To be conservative we could just go with STATUS_FAILED but I'm not sure. If we got with the 'non-success' idea, we may need to check for a minimum value due to longer running tasks which may still be in progress. We would not want this addition to the cleanup job cleaning up tasks which may still be in progress. diff -Nurp /usr/bin/retrace-server-cleanup /usr/bin/retrace-server-cleanup.bz1086321 --- /usr/bin/retrace-server-cleanup 2014-03-25 09:04:07.000000000 -0400 +++ /usr/bin/retrace-server-cleanup.bz1086321 2014-04-17 09:16:56.019129321 -0400 @@ -123,3 +123,21 @@ if __name__ == "__main__": if task.get_age() >= CONFIG["DeleteTaskAfter"]: log.write("Deleting old task %s\n" % filename) task.remove() + + if CONFIG["DeleteFailedTaskAfter"] > 0: + # clean up old failed tasks + try: + files = os.listdir(CONFIG["SaveDir"]) + except OSError, ex: + files = [] + log.write("Error listing task directory: %s\n" % ex) + + for filename in files: + try: + task = RetraceTask(filename) + except: + continue + + if task.get_age() >= CONFIG["DeleteNonSuccessTaskAfter"] and task.get_status == STATUS_FAILED: + log.write("Deleting old non-success task %s\n" % filename) + task.remove()
(In reply to Dave Wysochanski from comment #3) > Assuming Michal you're ok with the design of a second config variable, I > think we just need a similar section in retrace-server-cleanup to the > existing one for DeleteTaskAfter, something like the below. > > To be conservative we could just go with STATUS_FAILED but I'm not sure. If > we got with the 'non-success' idea, we may need to check for a minimum value > due to longer running tasks which may still be in progress. We would not > want this addition to the cleanup job cleaning up tasks which may still be > in progress. > > diff -Nurp /usr/bin/retrace-server-cleanup > /usr/bin/retrace-server-cleanup.bz1086321 > --- /usr/bin/retrace-server-cleanup 2014-03-25 09:04:07.000000000 -0400 > +++ /usr/bin/retrace-server-cleanup.bz1086321 2014-04-17 > 09:16:56.019129321 -0400 > @@ -123,3 +123,21 @@ if __name__ == "__main__": > if task.get_age() >= CONFIG["DeleteTaskAfter"]: > log.write("Deleting old task %s\n" % filename) > task.remove() > + > + if CONFIG["DeleteFailedTaskAfter"] > 0: > + # clean up old failed tasks > + try: > + files = os.listdir(CONFIG["SaveDir"]) > + except OSError, ex: > + files = [] > + log.write("Error listing task directory: %s\n" % ex) > + > + for filename in files: > + try: > + task = RetraceTask(filename) > + except: > + continue > + > + if task.get_age() >= CONFIG["DeleteNonSuccessTaskAfter"] > and task.get_status == STATUS_FAILED: > + log.write("Deleting old non-success task %s\n" % > filename) > + task.remove() The above patch does not even work but I will attach a cleaned up / tested patch soon.
Created attachment 887198 [details] Patch to add a new config variable and section to allow for more aggressive deletes of failed tasks
Test case / outline: 1. On retrace-server system, run script to find failed tasks (see attachment). The system should have at least one failed task which is newer than the value of DeleteTaskAfter 2. Add DeleteFailedTaskAfter to /etc/retrace-server.conf with a value just below the value of DeleteTaskAfter and such that it should remove the failed task 3. Run the cleanup script, look at the cleanup log, and make sure you see something like the below. On my system the log is /var/log/retrace-server/cleanup.log # tail /var/log/retrace-server/cleanup.log ... [2014-04-17 10:01:13] Running cleanup Deleting old failed task 217408639
Looks fixed in retrace-server-1.11-4.el6.noarch
retrace-server-1.12-2.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/retrace-server-1.12-2.el6
retrace-server-1.12-2.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.