Description of problem: Suggestion is to have a separate machine for logs. Instead of pushing all logs to the Scheduler we would store them in the labs and point back to them from the Scheduler.
Update lab controller/proxy to cache active recipe logs on the lab controller itself. cron job run daily would push all logs for finished recipes to results server. This can be either the scheduler or another server for hosting results. (determined via config option) pseudo code for recipe in recipes_finished_but_not_migrated: rsync recipe to result server update DB location for logs delete recipe logs from cache