Bug 1125439

Created https://github.com/rhq-project/rhq/pull/105 with potentially useful changes related to this issue.

The multiple baselines come about due to overlapping data purge jobs being run, and within the data purge jobs unfortunate timing can produce overlapping baseline generation.  In general stateful quartz jobs don't overlap, that's exactly the point of declaring them stateful.  But it is possible in  basically one scenario.  First, it needs to be an HA environment, which means multiple servers and therefore muiltiple (clustered) quartz schedulers. Then, it requires a false positive check for a "down" scheduler.  Each quartz scheduler intermittently checks in with the DB and updates a check-in time.  When it does that it also checks the other schedulers to make sure they have all checked in on schedule.  If a scheduler has missed a check-in it it considered down and the scheduler checking-in performs recovery logic on in-progress jobs which it presumes are dead.  It:
 1) cleans up the in-progress job
 2) if the job is set for recovery it is then executed immediately (although
    all of our jobs are set to recovery=false)
 3) it then applies the misfire action set on the trigger.
    By default (for a cron trigger) it will immediately launch the job if it has
    misfired at least once. A "misfire" means it missed a scheduled execution
    because a previous execution was still running.  If it has not misfired
    it is just rescheduled for the next scheduled execution after the current
    time.

This is what is happening. A running scheduler is missing it's check-in. And therefore, the job is actually running but it is considered dead.  The recovery ends up executing it again, and then you have overlapping executions.

There are basically 3 reasons this happens.  By far the most prevalent is lack of clock sync among the servers and/or database.  It is critical that clock sync is close, quartz only allows a little bit of wiggle room when checking for a missed check-in.  The other main reason is some condition that doesn't allow nodes to "check in" to the database in a timely manner (at the cluster check-in interval).  This can happen if there is too much lag time when updating the DB, for whatever reason, potentially only on one server.

Master commit a06323a067fa942bb5100be2845c47f514626c79
Author: Jay Shaughnessy <jshaughn>
Date:   Sat Aug 16 14:59:09 2014 -0400

 Due to some git trouble this is something of a manually generated squash
 commit.  The changes are somewhat two-fold.  First, there are changes
 relevant to the initially reported symptom, which was the existence of
 duplicate baselines for a single schedule.  The code looked like it was
 capable of generating a storing a duplicate, due to the use of Lists (not
 Sets) in the APIs, and also the lack of a unique index even though there is
 a 1:1 relationship between baseline and schedule.  The first set of commits
 addressed these issues and also adding chunking to avoid large Txs and also
 oracle IN-clause limits.
 - Make the db index on baseline's scheduleId unique. There should only be
   one baseline for a schedule. This will prevent the problem in the BZ,
   possibly at the expense of failure when trying to store a duplicate, but
   that would perhaps help us pinpoint a problematic code-path.
 - Use Sets (or a Map) as opposed to Lists in several places in the local API.
 - Simplify and I think make more efficient the query for selecting schedules
   without a current baseline. Make it return distinct ids (this is a hedge,
   I don't think it would return duplicates, but this ensures that constraint).
   Replace a left join with a NOT EXISTS sub select.
 - Chunk the storage of baselines to prevent large Txs, don't pass entities
   across Tx boundary, fetch all Schedules in one query, and (maybe) simplify
   the entity manager interaction.
 - Simplify some of the TransactionAttribute decls
 - Mark MeasurementSchedule.baseline as an optional 1-1 relationship, just to
   make it explicit on the entity.
 - Fix some local SLSB jdoc

 Later, the actual root cause was discovered: the baseline job executing
 concurrently and the same schedule occurring in the overlapping jobs. This
 scenario was originally thought to be impossible  due to the stateful
 nature of the quartz job. But in fact it is possible in certain conditions
 due to quartz clustered recovery logic(see BZ comments).  These changes
 minimize the concurrent job possibility:
 - increase the quartz clusterCheckinInterval from 30s to 2m.  This
   decreases the chance of false positives when quartz checks for down
   schedulers and should have little or no effect on the recovery of our jobs.
 - move the data calculation work from DataPurgeJob to the new DataCalcJob,
   leaving DataPurgeJob to deal only with RDB maintenance and letting
   the new DataCalcJob only work on baseline and OOB generation.  There is
   no reason for DataCalc work to have to wait for DB maintenance work.
   Also, stagger the jobs by 1/2 hour to lessen concurrent work.
 - for the cron jobs, switch the misfire policy from the default (RUN_ONCE)
   to DO_NOTHING.  So, instead of long running jobs again starting up
   immediately after finishing, let it wait for the next scheduled start
   time.  This also reduces the chances of overlap if we do in fact get a
   false positive recovery situation.  (as an aside, there is a good web page
   here regarding clustered quartz logic:
     http://flylib.com/books/en/2.65.1.91/1/)

 Given the work described above it's now very unlikely but not completely
 guaranteed that we won't have overlapping baseline calculation jobs.
 Because we now have a unique index on baseline.schedule we will immediately
 detect an attempt to store a duplicate baseline for a schedule.  In the
 unlikely event that this happens we know it's [very likely] due to an
 overlapping job. So if we fail to save baselines because of a constraint
 violation we just catch, log a warning and continue.

release/jon3.3.x commit 184f13babbfeacc5bb0e215f2c0145ba9197aba2
Author: Jay Shaughnessy <jshaughn>
Date:   Sat Aug 16 14:59:09 2014 -0400

 Cherry-pick master: 3e4f9571733ea0c7867acf693efc5fc82110b41a

Moving to ON_QA as available for test with the following brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=381194

@Jay: hi, need some info please if it would fix/correct the situation from old release (so if one does `rhqctl upgrade` the servers with such situation)? (may i suspect - no)?

Another thing - any concrete scenario for making a bit chance to make such a situation in 3.3 ? :)

Would it be enough making a HA 2 servers and plug some agents waiting for several rounds of purge jobs to run and trace the database for possible duplicates please?

So far I understand that with current enhancements we preventing it from being happen in many aspects - so my chances to try to reproduce or find a hole - would be close to 0 :)

thanks.

Hi Garik,

Yes, the upgrade will repair the duplicates issue.  When the DB is upgraded we execute statements that first remove duplicate baselines, and then apply a unique index that will prevent duplicates in the future.  We don't really expect duplicates given the other changes here but we can't completely rule it out.  But now we will handle it if it happens.

Recreation steps are not simple, it was hard *before* the fix, requiring some manual manipulation of the Quartz jobs.  I think running whatever HA and HA perf testy plans that you guys typically run through would be sufficient.

#VERIFIED

made a performance server with multiple agents and HA structure - all worked fine and no logs as such in description appeared after a day of running.