Bug 1337789
Summary: | beaker-log-delete left behind some job logs from May 2013 and earlier | ||
---|---|---|---|
Product: | [Retired] Beaker | Reporter: | matt jia <mjia> |
Component: | general | Assignee: | Dan Callaghan <dcallagh> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Matt Tyson 🤬 <mtyson> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 22 | CC: | bmcivor, dcallagh, huiwang, jorris, mjia, mtyson, rjoost |
Target Milestone: | 25.0 | Keywords: | FutureFeature, Patch, Triaged |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-03-19 04:19:07 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
matt jia
2016-05-20 05:29:24 UTC
(In reply to Jon Orris from comment #1) > https://gerrit.beaker-project.org/#/c/5245/ Pushed to wrong branch. Correct change is: https://gerrit.beaker-project.org/#/c/5250/ This bug fix is included in beaker-server-24.0.git.210.01bf4bf which is currently available for testing here: https://beaker-devel.app.eng.bos.redhat.com/ Matt, Jon: has someone actually checked that all the log_* rows we will be deleting here have already had the log files deleted on disk as well? Otherwise this change is actually going to make things worse, not better (since now we will have a pile of old logs left behind on disk with no reference to them in the database to locate them). In that case the proper fix would be to figure out why beaker-log-delete has not already deleted these files and their corresponding log_* rows. But in the meantime we need to roll this patch out of 24.0, right? (In reply to Dan Callaghan from comment #7) > But in the meantime we need to roll this patch out of 24.0, right? Yeah, I have a patch to revert the patch. https://gerrit.beaker-project.org/#/c/5572/ I'm actually putting this one back to NEW, since with the recent team re-shuffling it highly likely that someone else needs to finish it. beaker-log-delete and the related model code in Job.delete() has received a lot of bug fixes over the years since it was originally introduced in 2011, so it might not be worthwhile digging through that to find exactly why it ended up leaving some old jobs in this state: where the job is marked as deleted, but some logs are still left behind. I did look out for old code which looked like it might be failing to delete the logs but then still committing the db transaction -- but actually, all the fixes I could see seemed more likely to be the other way around -- that is the log files would be deleted but then the db transaction *not* committed. Regardless, we probably just need to add some code to beaker-log-delete to look for jobs which are marked as deleted, but still have logs recorded against them, and to delete those again "properly". It would be good to get this done sooner rather than later, since if there really are 20 million log files which could be deleted this would potentially save quite a bit of storage. So this mess with "deleted" vs. "soft-deleted" vs. "to_delete" has been bugging me for years... It might be finally time to clean that up: https://gerrit.beaker-project.org/5765 remove Job "sanitise" methods https://gerrit.beaker-project.org/5766 remove unused params on Job.find_jobs https://gerrit.beaker-project.org/5767 use Job.is_deleted in queries https://gerrit.beaker-project.org/5768 rationalise Job.delete_jobs, Job._delete_criteria, job.delete_jobs https://gerrit.beaker-project.org/5769 clarify deleted vs. expired vs. purged for jobs With that clean-up in place, the problem here is a lot easier to describe: we have some old jobs which have been *deleted* but not successfully *purged*. And therefore we can just update the db to clear the purged timestamp and let beaker-log-delete purge them again. https://gerrit.beaker-project.org/5770 make beaker-log-delete fix up old jobs which still have logs Beaker 25.0 has been released. Release notes are available upstream: https://beaker-project.org/docs/whats-new/release-25.html |