Bug 834019

Jay S. did something in the availability scheduling that was similar - at least the end goal was similar and that is to spread out the initial schedule so things aren't scheduled to be collected at roughly the same time. Need to look at the code Jay did for that and perhaps see if the same method can be used here.

Jay noted that for this to be efficient, we need to group metric requests for the same resource. This is indeed a requirement for us because we have a custom plugin that fetches the measurements for all metrics of a given resource in a single remote call. This requirement should be easy to satisfy by calculating the shift based on a hash function (e.g. modulo) of the resource ID.

However, there is another problem: in our case, all resources are highly available, which means that they are part of compatible groups. In that case it is not desirable that the measurement schedules for the different members of a given resource group are shifted randomly with respect to each other. Instead, measurements should be taken at the same time. Therefore it is probably better to use a hash function of the resource key instead of the resource ID.

Finally, I think that the first scheduled measurement should not be calculated relative to the agent start time, but relative to an absolute point in time (e.g. the UNIX epoch). This allows to satisfy the last requirement even if the measurements for the resources in a group are taken by different agents (which will likely be the case in most RHQ setups). It also helps satisfy the first requirement when metrics are enabled at runtime.

See also Bug 783603

Interesting, so the attached patch is small and relatively non-intrusive. If I read this right, all this is doing is taking a measurement schedule that was late and simply punting a little bit more into the future - essentially delaying the next collection 30s into the future.  The difference here is the old code was more conservative and punted until the next schedule collection interval (which could be 10m in the future or some such number) giving the agent more time to catch up doing things (which may or may not be due to measurement collection if the agent is CPU starved for other reasons). Punting only for another 30s, in general, may not be enough time to let the agent try to settle down in a steady state.

What if, rather than punt 30s in the future, we punt (next collection time + 30s) - essentially spreading out the next collection further out in the future than it would otherwise by 30 seconds?  Thus, the earlier description of the attached patch can be slightly modified to say:

> The idea is that instead of rescheduling the measurement according to the
> original schedule (e.g. from [0s,30s] to [10m,10m+30s]), it should simply
> be rescheduled to the next interval (from [0s,30s] to ***[10m+30s,10m+60s]***).

The *** *** part is what I propose we change from the attached patch.

> Finally, I think that the first scheduled measurement should not be calculated
> relative to the agent start time, but relative to an absolute point in time
> (e.g. the UNIX epoch).

This part scares me and before we do anything like this, we'd need some testing and prototyping because now this potentially means that many collections across all agents won't necessarily be "randomly spread out" but will more likely overlap. How does this affect the amount of data from all agents that get into the server(s) - will the servers be bombarded by a synchronocity of cyclical agent measurement reports with a period of relative inactivity in between? I'm just not sure I fully understand the ramifications of normalizing all collection start times to epoch millis and until we do understand the ramifications on system behavior, we should err on the side of caution.

I committed to master this: cd6c2fa4ce4f128f80145438499dbfc1223fac42

A later commit added some code to our base arquillian framework and I committed a stubbed out unit test method that is supposed to test the fix for this BZ. I will work on fleshing out that test next. But I needed the framework code first. See commit: 5e823b969b8cc830432cfc2e055ed3292464c4d8

finished the unit test - committed to master: 00db03b19d5a70e9798405b33f91ed150e38382e

I'm not sure how QE can test this. This was hard to get to replicate in unit test code! So I would say QE should just rely on the results of the unit test class LateMeasurementRescheduleTest.

Setting this back to ON_DEV.

The fix today is OK but after discussing with Mazz we think it can be improved. The issue with the fix today is that when metric collection falls behind, there may be a large number of schedule requests still remaining for that time bucket.  And all of them are then rescheduled for the same time in the future (collectionInterval + 31s). It is good that they are scheduled away from the collectionInterval, but more randomization would be better, as would rescheduling for a time that is potentially nearer to the current time.

The change will be to reschedule to 31s + randomSeconds(0..collectionInterval).

The fixed 31s ensures that we push out at least to some degree, allowing the agent some time to catch up and, I believe, this plays into our internal eval periods.  But then introduce more randomization to better spread out collection requests, and also don't, in general, wait so long to retry, because this could cause the initial collection to still take an unacceptably long time.

Also, add more debug and trace level logging for better diagnostics in this area.

A little more revision after getting into this in detail. Importantly, the rescheduling of late collection requests is:

  *Now* + 30s + [1..collectionInterval]

using "Now" (i.e. current system time) as opposed to request.getNextCollection() (i.e. the originally scheduled time) gives us more predictability about the range in which the request will be scheduled.


commit e09fb3b66648a2617825657e8ed153d98b0cae78
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Nov 14 16:39:05 2012 -0500

    Revisit the solution for this BZ and improve by adding more
    randomization to the reschedule times.
    - update test accordingly
    - add some debug and trace level debugging around late
      collection handling.

commit bcce31290b99816d9a82ad1b6df70c187c563ec2
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Nov 14 17:14:31 2012 -0500

    trivial - enhance logging slightly

Bulk closing of items that are on_qa and in old RHQ releases, which are out for a long time and where the issue has not been re-opened since.