Bug 629025 - Implement a cap on size and number of files uploaded
Summary: Implement a cap on size and number of files uploaded
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Retired
Component: lab controller
Version: 0.5
Hardware: All
OS: Linux
high
medium
Target Milestone: ---
Assignee: Bill Peck
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 626331 632609
TreeView+ depends on / blocked
 
Reported: 2010-08-31 18:00 UTC by Bill Peck
Modified: 2019-05-22 13:37 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-02-10 01:01:01 UTC
Embargoed:


Attachments (Terms of Use)

Description Bill Peck 2010-08-31 18:00:00 UTC
Description of problem:

The test machines can easily overwhelm the scheduler with data.  We should implement a limit on the number of files and the size of the files that can be uploaded per recipe/task.

The limits and behaviour should be configurable via the harness config file.

resultfilesizelimit=100MB
resultfilelimit=10
enforcement=warn

enforcement would be either warn or limit.  warn would record a warning in the tasks output so that the test writer knows they hit the limit but still allow the file to grow.  limit would also record the same result but not allow the file to grow anymore.

Comment 1 Kevin Baker 2010-08-31 20:33:43 UTC
This solution should also keep a record of all those recipes(jobs?) which hit the soft or hard limit.

Jumping to a possible implementation, I was thinking that when the harness detects the limit being exceeded that it send a message to the Scheduler for it to create a database record of it. The db record should have enough information to generate interesting stats on which jobs/recipes/tasks/tests/users are generating too much data. This will be immensely useful in determining solutions.

Comment 2 Marian Csontos 2010-09-07 12:49:18 UTC
Taking this with high-priority as this is responsible for server overload.

Comment 3 Marian Csontos 2010-09-16 09:18:11 UTC
= The reporting part =

Recipe/job limits/statistics does not seem much useful to me: it's all down to tasks. Recipe witch some 200 tasks (e.g. Tier1 tests) will generate approximately 100-times more files/data than recipe with 2 tasks. And these tasks are often weakly related only.

Re: Comment 1: We do not need harness to get that: simple combining du with DB would do better: e.g. let's add recursive size (Task|Recipe|Job)'s log directory to \1's record in DB.

Then we could aggregate (e.g. average) on Tasks.

I would like to see:

- the most greedy tasks
- the task runs which surpasses max(M, task's previous average by factor of N) which should raise a suspicion

= The harness part =

This should prevent LC/Scheduler overload: I want it to raise warning after reaching certain level and optionally stop the upload(s) after reaching the critical level.

And tasks should be able to opt-out/change the limit: there may be (understand "will be") a legal use-case for task uploading bigger/more files, similar how the watchdog is adapted:

The complex task doing what would take 50 simpler tasks, will have 50x more data than simpler ones.

Comment 4 Marian Csontos 2010-10-12 19:12:06 UTC
I have tested new harness and fixed few minor glitches. Everything should be fine now and after next batch succeeds I will make a package for update.

Comment 5 Marian Csontos 2010-10-12 19:17:40 UTC
Bill, hare are the new config.options:

http://git.fedorahosted.org/git/?p=beah.git;a=blobdiff;f=beah_beaker.conf;h=8c5a393a501d6c9e0ab5b1df4233b1d7315574d3;hp=b3aa86f6ffbd8b427c9695f7ff4a0017334c9268;hb=25759df44d7b1fa21d49fb8ca62afd4e9c133f41;hpb=95df65b8d2895f19af5276f240ec7db0abac7985

I have forgotten, but this will need a beah_beaker.conf file modification (which is built in ks now.)

Limits can be set on (bytes uploaded, file size or file count) per (file, task or recipe)

Comment 6 Bill Peck 2010-10-15 16:11:06 UTC
I've asked eng-ops for suggested settings for these values.  once I get that I'll push these changes into beaker-lab-controller

Comment 7 Marian Csontos 2010-11-08 09:06:14 UTC
Updating milestone to keep it visible.

Comment 8 Marian Csontos 2011-01-25 10:35:32 UTC
Bill, could we move this forward?

For now even an unreasonable big upper limits would be fine.

One of my tests got into loop the other day [ not by my fault, of course ;-) ] and uploaded few gigs to production server. I had eng-ops deleting the files, but it had still caused some load...

What about starting with hard limits 100MB per file and 1GiB per task? Those values are pretty liberal IMO as I want to keep disturbance low.

Let's set soft limits at 1/4 of hard limits to let people know they may be doing something not the right way.

We may also want to make the limits tighter gradually...

After a month or so, we could divide those configuration options by 2 and wait for another month or so. And iterate, until we got things into shape.

Comment 9 Bill Peck 2011-01-26 21:34:01 UTC
mcsontos,

I've requested input on sane defaults.

Comment 12 Dan Callaghan 2011-02-10 01:01:01 UTC
Beaker 0.6.4 has been released.


Note You need to log in before you can comment on or make changes to this bug.