Description of problem: Attempt to delete huge number of partition events (more then 1 million events in the database) fails with error: ******************************* 10:29:07,685 ERROR [org.jboss.as.controller.management-operation] (http-/0.0.0.0:7080-72) JBAS014612: Operation ("read-attribute") failed - address: ([("deployment" => "rhq.ear")]): java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256) [rt.jar:1.8.0_66] at java.util.HashMap.putVal(HashMap.java:630) [rt.jar:1.8.0_66] at java.util.HashMap.put(HashMap.java:611) [rt.jar:1.8.0_66] at java.util.HashSet.add(HashSet.java:219) [rt.jar:1.8.0_66] at org.jboss.as.controller.registry.AbstractModelResource.getChildren(AbstractModelResource.java:115) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:680) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.OperationContextImpl.readResource(OperationContextImpl.java:628) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecuteInternal(ReadAttributeHandler.java:153) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecute(ReadAttributeHandler.java:96) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:249) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:563) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:283) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA] at org.rhq.common.jbossas.client.controller.DeploymentJBossASClient.getDeploymentPath(DeploymentJBossASClient.java:99) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA] at org.rhq.enterprise.server.core.CoreServer.getEarDeploymentDir(CoreServer.java:187) [rhq-server.jar:4.12.0.JON330GA-redhat-4] at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) [:1.8.0_66] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66] at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66] at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [rt.jar:1.8.0_66] at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) [:1.8.0_66] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66] at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66] at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) [rt.jar:1.8.0_66] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [rt.jar:1.8.0_66] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [rt.jar:1.8.0_66] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [rt.jar:1.8.0_66] ******************************* Version-Release number of selected component (if applicable): JBoss ON 3.3.4 How reproducible: Always Steps to Reproduce: 1. Make sure that JBoss ON database - rhq_partition_event table contains more then a million events. 2. Attempt to delete partition events by clicking on "Purge All" button in JBoss ON UI -> Administration -> Partition Events page; 3. Actual results: Deletion of partition events fails. Expected results: Partition events are properly deleted. Additional info: "Purge All" - delete of partition events should be done using smaller batches.
Duplicate of BZ 1255196 (fixed in 3.3.5)
@Michael, I do not think the fix for Bug 1255196 addresses this issue. Looking at the code, I can see we load all events into memory[1] when attempting a purge of all partition events. I would expect that paging should be used or some other method should be used to perform the purge. [1]: https://github.com/rhq-project/rhq/blob/release/jon3.3.x/modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/cloud/PartitionEventManagerBean.java#L159
Right, the customer should use automatic cleanup of partition events (enabled from system settings). For that, the 3.3.5 fixes the out of memory issues as it does not load everything to memory. At least that seems to be wish in the customer ticket to keep it clean.
@Michael, is the automatic cleanup of partition events a new system setting? I do not see that in version 3.3.4
The Advanced Purge Settings tab is not in JBoss ON 3.3.4 only RHQ 4.13 how can this be added to the Jboss ON release?
@dsteigne, it cannot. This is not supported in the product and is a community only feature. This issue will require a fix as described in comment 2.
@Larry, ok thank you.
Fixed in the master: commit 2a589c3867f931d00845c7f3c303d0d3197b96be Author: Michael Burman <miburman> Date: Tue Jan 26 15:25:55 2016 +0200 [BZ 1297702] Change purgeAllEvents to use PurgeManager to avoid OutOfMemory issues
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Moving to ON_QA as available to test with JON 3.3.6 DR01 brew build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=499890
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-1519.html