When we have very large environments, it may be that we cannot process avail reports fast enough before our checkForSuspectAgents job backfills the agent.  I see this every now and again - where resources go red then green and ping pong back and forth between up and down.

We should be able to configuration the "quiet time" that we allow before backfilling.  Currently, its hardcoded to 2 minutes and the agent sends the avail messages every 1 minute.  The agent is configurable - set "rhq.agent.plugins.availability-scan.period-secs" to have it send avail reports faster or slower than 1 minute.  We currently have a hack to configure the server (set a prop in rhq-server.properties - see below).  We should either consider putting a value in RHQ_SYSTEM_CONFIG so its the same across the cloud (and changable via Admin UI page) or we could put some kind of smarts in the server so it could say something like, "I'm getting clobbered with alot of agent messages/inventory reports/etc - I'll let agents slide another 2 minutes on avail reports - so I won't backfill unless I don't hear from an agent in 4 minutes".  The server could then readjust later when it catches up, back to the 2-minute backfill quiet time.

In our AgentManagerBean we have:

    @SuppressWarnings("unchecked")
    public void checkForSuspectAgents() {
        if (log.isDebugEnabled())
            log.debug("Checking to see if there are agents that we suspect are down...");

        // TODO [mazz]: make this configurable via SystemManager bean
        long maximumQuietTimeAllowed = 1000L * 60 * 2;
        try {
            String propStr = System.getProperty("rhq.server.agent-max-quiet-time-allowed");
            if (propStr != null) {
                maximumQuietTimeAllowed = Long.parseLong(propStr);
            }
        } catch (Exception e) {
        }