Bug 1579733

Description of problem:
A large number of unused alert definitions (8000+) cannot be purged during Data Purge Job that runs every hour. Initially, a large number of alerts were triggered frequently, so the transaction that is used to remove alert definitions would timeout (this was a case even when the transaction timeout was changed to 5 hours). So, in order to remove unused alert definitions, the following steps were performed:

1. Disabled all currently enabled alerts;
2. The JBoss ON server started in the maintenance mode (in order to prevent JBoss ON agents to connect);
3. HornetQ messaging folders emptied ( $JON_SERVER_HOME/jbossas/standalone/data/messagingbindings...);
4. The leftover transaction data removed by emptying $JON_SERVER_HOME/jbossas/standalone/data/tx-object-store; 
5. Changed the transaction timeout to 10h;
6. Started JBoss ON server;

The server.log file showed that 3 hours after the Alert definition unused purge method was started, the server started to show timeout issues: management plug-in requests were timing out, many EJB timer timeouts and many Quartz timer timeouts. Finally, the server.log file shows:  java.lang.OutOfMemoryError: GC overhead limit exceeded errors. 


Version-Release number of selected component (if applicable):
JBoss ON 3.3.10

How reproducible:
Sometimes

Steps to Reproduce:

The following are the steps that I performed in order to get transaction timeout when trying to remove unused alert definitions. However, I couldn't reproduce GC overhead limit exceeded errors as that's in the customer's log file.

1. I created a large number of alert definitions in my JBoss ON server using REST API:

****************************************************
while true; do  curl -s -L -H 'Content-Type: application/json' -X POST http://bkramer.usersys.redhat.com:7080/rest/alert/definitions?resourceId=18306 -d '{"name":"restPlatformMeas","enabled":true,"priority":"MEDIUM","conditionMode":"ANY","conditions":[{"category":"CHANGE","id":37824,"threshold":null,"option":null,"triggerId":null,"comparator":null,"measurementDefinition":10019,"name":"Actual Free Memory"}],"notifications":[{"id":10041,"senderName":"Direct Emails","config":{"emailAddress":"bkramer"},"extraConfig":{}}],"dampeningCategory":"NONE"}' -u rhqadmin:rhqadmin; sleep 1s; done
****************************************************

2. Above metric "Actual Free Memory" is enabled to be collected every 30 seconds (very aggressive);

3. Once the number of created alert definitions is greater then 8000 and the number of messages in the AlertConditionQueue is > 400k, continue to create new alert definitions but delete all alert definitions that were previously created (using JBoss ON CLI [1]) and also remove all previously triggered alerts (from the alert history page in the JBoss ON UI). 

4. Execute the following SQL to make sure that the number of unused alert definitions > 8000:

****************************************************
select count(alertdefin0_.ID) as col_0_0_ from RHQ_ALERT_DEFINITION alertdefin0_ where alertdefin0_.DELETED=true and (alertdefin0_.ID not in  (select alertdefin2_.ID from RHQ_ALERT alert1_ inner join RHQ_ALERT_DEFINITION alertdefin2_ on alert1_.ALERT_DEFINITION_ID=alertdefin2_.ID))
****************************************************

4. Wait for the DataPurgeJob and Alert definition unused purge method to start;


Actual results:
In my environment, removal of the alert definitions failed due to the transaction timeout. In the customer's environment, deletion of the alert definitions fails with java.lang.OutOfMemoryError: GC overhead limit exceeded errors although their JBoss ON server had 10Gb allocated. 


Expected results:
Unused alert definitions are properly deleted and no exceptions are triggered. 


Additional info:

[1] 
****************************************************
var criteria = new AlertDefinitionCriteria();

//Change the number of entries returned by search command to 1000 instead of default 200 - by using setPaging method
criteria.setPaging(0,10000);
criteria.addFilterName("restPlatformMeas");

var alertdefs = AlertDefinitionManager.findAlertDefinitionsByCriteria(criteria);

if ( alertdefs != null ) {
  if ( alertdefs.size() > 0 ) {
     for ( i =0; i < alertdefs.size(); ++i) {
         var alert = alertdefs.get(i);
         println("alert: " + alert);
         AlertDefinitionManager.removeAlertDefinitions([alert.id]);
     }
  }
}
****************************************************