Bug 1063430

Candidate for 3.2.x. Without this fix OOB calculations should be considered broken for all practical purposes. 

+++ This bug was initially created as a clone of Bug #1059412 +++

Description of problem:

See: https://community.jboss.org/message/855449#855449

"17:49:03,425 INFO  [org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean] (RHQScheduler_Worker-4) Finished calculating 32 OOBs in 3086252 ms  
17:49:03,425 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-4) Auto-calculation of OOBs completed in [3086260]ms  

(This is roughly 51 minutes, not good.)

I'm not sure why OOB calculations would take this long. I'm digging into why but if somebody knows what to look for here, that'd be awesome.
 
I'm thinking it'd be nice to turn off OOB stuff as I don't use it anyway, and it looks like it's making my purge job take more than an hour anyway.
 
Seems to me it's just the number of metrics is too much, that querying Oracle for each one takes more time than it can support. It seems the answer might be to:
1) Have an option to disable OOB completely
2) Only do a partial calculation (like 30%)
3) Optimize the round trip to the database, e.g. query all (or a portion) baselines at once, then merge the results.
"

Version-Release number of selected component (if applicable): 4.9


How reproducible: Always, with enough schedules


Additional info:

Should be possible to optimize this. Or simply disable OOB calculations. This impacts data purge.

--- Additional comment from Heiko W. Rupp on 2014-02-04 07:44:50 EST ---

It is possible to disable automatic baseline calculation by setting the frequency to 0. 
So here we could check that if baseline calculation is disabled, also no baselines are calculated, as they would need the baseline high/low as reference points.

It is still possible to have baselines though (e.g. via REST-api) so we can't just completely switch off OOB computation if baselineFrequency == 0.

One option could be to check in case baselineFrequency==0 for Baselines in the Baseline table and then only feed those metrics into 
org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean#calculateOOB instead of all metrics (basically filtering the list passed in into org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean#computeOOBsForLastHour )

Also it should be possible from #computeOOBsForLastHour to just get all existing baselines in one DB-roundtrip and then passing the relevant value into #calculateOOB instead of having all the EM calls inside #calculateOOB 
(and the query #calculateOOB should be a NamedQuery so that it can be prepared in advance)

--- Additional comment from Heiko W. Rupp on 2014-02-05 02:33:03 EST ---

The attached patch reduces the run time for me from 

17:06:50,501 INFO  [org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean] (RHQScheduler_Worker-3) Finished calculating 1223 OOBs in 116039 ms

to

05:04:39,678 INFO  [org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean] (RHQScheduler_Worker-4) Finished calculating 1213 OOBs in 16359 ms

The patch probably needs a little more work with respect to the signature change detector.

--- Additional comment from John Sanda on 2014-02-10 11:37:50 EST ---

The patch has been tested and applied to master.

master commit hash: 3b9bbfd4355