Candidate for 3.2.x. Without this fix OOB calculations should be considered broken for all practical purposes. +++ This bug was initially created as a clone of Bug #1059412 +++ Description of problem: See: https://community.jboss.org/message/855449#855449 "17:49:03,425 INFO [org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean] (RHQScheduler_Worker-4) Finished calculating 32 OOBs in 3086252 ms 17:49:03,425 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-4) Auto-calculation of OOBs completed in [3086260]ms (This is roughly 51 minutes, not good.) I'm not sure why OOB calculations would take this long. I'm digging into why but if somebody knows what to look for here, that'd be awesome. I'm thinking it'd be nice to turn off OOB stuff as I don't use it anyway, and it looks like it's making my purge job take more than an hour anyway. Seems to me it's just the number of metrics is too much, that querying Oracle for each one takes more time than it can support. It seems the answer might be to: 1) Have an option to disable OOB completely 2) Only do a partial calculation (like 30%) 3) Optimize the round trip to the database, e.g. query all (or a portion) baselines at once, then merge the results. " Version-Release number of selected component (if applicable): 4.9 How reproducible: Always, with enough schedules Additional info: Should be possible to optimize this. Or simply disable OOB calculations. This impacts data purge. --- Additional comment from Heiko W. Rupp on 2014-02-04 07:44:50 EST --- It is possible to disable automatic baseline calculation by setting the frequency to 0. So here we could check that if baseline calculation is disabled, also no baselines are calculated, as they would need the baseline high/low as reference points. It is still possible to have baselines though (e.g. via REST-api) so we can't just completely switch off OOB computation if baselineFrequency == 0. One option could be to check in case baselineFrequency==0 for Baselines in the Baseline table and then only feed those metrics into org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean#calculateOOB instead of all metrics (basically filtering the list passed in into org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean#computeOOBsForLastHour ) Also it should be possible from #computeOOBsForLastHour to just get all existing baselines in one DB-roundtrip and then passing the relevant value into #calculateOOB instead of having all the EM calls inside #calculateOOB (and the query #calculateOOB should be a NamedQuery so that it can be prepared in advance) --- Additional comment from Heiko W. Rupp on 2014-02-05 02:33:03 EST --- The attached patch reduces the run time for me from 17:06:50,501 INFO [org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean] (RHQScheduler_Worker-3) Finished calculating 1223 OOBs in 116039 ms to 05:04:39,678 INFO [org.rhq.enterprise.server.measurement.MeasurementOOBManagerBean] (RHQScheduler_Worker-4) Finished calculating 1213 OOBs in 16359 ms The patch probably needs a little more work with respect to the signature change detector. --- Additional comment from John Sanda on 2014-02-10 11:37:50 EST --- The patch has been tested and applied to master. master commit hash: 3b9bbfd4355
commit 6eb9c50486316610f7874040f97c64d8ba8cd660 Author: John Sanda <jsanda> Date: Sun Feb 9 20:00:20 2014 -0500 [BZ 1059412] Applying patch to speed up OOB calculations (cherry picked from commit 3b9bbfd4355f466f128074f603d0dab48d1766d5) Signed-off-by: Lukas Krejci <lkrejci>
( That got cherry-picked on 06-21 and is thus in dr3)
Moving to ON_QA as available for test in latest build: http://jon01.mw.lab.eng.bos.redhat.com:8042/dist/release/jon/3.2.2.GA/6-28-2014/
# COMMENT taking for verification.
# COMMENT scenario to perform: 1. setup a JON 3.2.0 GA server with several agents (to fill up statistics fast) 2. wait for some period of time to make some data available 3. grep logs to collect: `cat ~/jon-server-3.2.0.GA/logs/server.log | grep "Auto-calculation of OOBs completed in "` 4. upgrade to JON 3.2.2 DR3 5. wait some more time 6. grep again compare performance.
# VERIFIED JON 3.2.2 DR4 has the improvement: it decreased for me (with 5 agents) from ~15 sec. to 4 sec time. === 00:00:43,971 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-3) Auto-calculation of OOBs completed in [13867]ms 01:00:26,189 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-2) Auto-calculation of OOBs completed in [14221]ms 02:00:25,705 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-5) Auto-calculation of OOBs completed in [14256]ms 03:00:27,403 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-4) Auto-calculation of OOBs completed in [15425]ms 04:00:31,674 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-5) Auto-calculation of OOBs completed in [6701]ms 05:00:20,577 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-3) Auto-calculation of OOBs completed in [6454]ms 06:00:41,183 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-2) Auto-calculation of OOBs completed in [5072]ms 07:00:21,776 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-1) Auto-calculation of OOBs completed in [4025]ms 08:00:44,999 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-5) Auto-calculation of OOBs completed in [2902]ms 09:00:17,677 INFO [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-2) Auto-calculation of OOBs completed in [3417]ms ===
This has been verified and released in Red Hat JBoss Operations Network 3.2 Update 02 (3.2.2) available from the Red Hat Customer Portal[1]. [1]: https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=31783