Bug 1293011

Summary: beaker-log-delete takes an unreasonable amount of CPU and memory to delete very large jobs
Product: [Retired] Beaker Reporter: Dan Callaghan <dcallagh>
Component: generalAssignee: Roman Joost <rjoost>
Status: CLOSED CURRENTRELEASE QA Contact: tools-bugs <tools-bugs>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 21CC: dcallagh, dowang, mjia, rjoost
Target Milestone: 23.0Keywords: Patch
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-07 23:09:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Callaghan 2015-12-19 02:57:36 UTC 
Related to bug 1293007, we are trying to recover from a job which contained two recipes which accidentally reported a very large number of results in an infinite loop (31457 total results in each recipe, 63543 total result logs in each recipe). beaker-log-delete struggles to delete such a large job, because it has load the entire job object graph into memory, including N db roundtrips for each result's logs. For example, right now it has spent 49 CPU minutes and consumed 1511MB virtual, 361MB resident in trying to load the large job and I'm not sure how much longer until it will need...

beaker-log-delete does allow applying a limit on the total number of jobs to delete in one run, but that doesn't help when there is a single job that is very large.

It needs to be smarter about loading the logs, ideally in batches, without requiring the entire job object graph to be fetched from the database. Bonus points if it can query all the logs in O(1) instead of O(N) db roundtrips.
Comment 1 Roman Joost 2016-01-08 01:49:17 UTC 
First patch is up:

https://gerrit.beaker-project.org/#/c/4574/1
Comment 2 Roman Joost 2016-03-23 00:30:27 UTC 
Dan has fixed regressions found when working on this and his patch on gerrit was merged:

https://gerrit.beaker-project.org/#/c/4719/
Comment 5 Dan Callaghan 2016-07-07 23:09:53 UTC 
Beaker 23.0 has been released.