Bug 1297702
| Summary: | Deletion of partition events in JBoss ON results in OutOfMemoryError when there is a million or more partition events to be deleted | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Operations Network | Reporter: | bkramer <bkramer> |
| Component: | Database | Assignee: | Michael Burman <miburman> |
| Status: | CLOSED ERRATA | QA Contact: | Filip Brychta <fbrychta> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | JON 3.3.4 | CC: | dsteigne, fbrychta, loleary, spinder |
| Target Milestone: | DR01 | Keywords: | Triaged |
| Target Release: | JON 3.3.6 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-07-27 15:32:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1297706 | ||
| Bug Blocks: | |||
Duplicate of BZ 1255196 (fixed in 3.3.5) @Michael, I do not think the fix for Bug 1255196 addresses this issue. Looking at the code, I can see we load all events into memory[1] when attempting a purge of all partition events. I would expect that paging should be used or some other method should be used to perform the purge. [1]: https://github.com/rhq-project/rhq/blob/release/jon3.3.x/modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/cloud/PartitionEventManagerBean.java#L159 Right, the customer should use automatic cleanup of partition events (enabled from system settings). For that, the 3.3.5 fixes the out of memory issues as it does not load everything to memory. At least that seems to be wish in the customer ticket to keep it clean. @Michael, is the automatic cleanup of partition events a new system setting? I do not see that in version 3.3.4 The Advanced Purge Settings tab is not in JBoss ON 3.3.4 only RHQ 4.13 how can this be added to the Jboss ON release? @dsteigne, it cannot. This is not supported in the product and is a community only feature. This issue will require a fix as described in comment 2. @Larry, ok thank you. Fixed in the master:
commit 2a589c3867f931d00845c7f3c303d0d3197b96be
Author: Michael Burman <miburman>
Date: Tue Jan 26 15:25:55 2016 +0200
[BZ 1297702] Change purgeAllEvents to use PurgeManager to avoid OutOfMemory issues
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions Moving to ON_QA as available to test with JON 3.3.6 DR01 brew build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=499890 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-1519.html |
Description of problem: Attempt to delete huge number of partition events (more then 1 million events in the database) fails with error: ******************************* 10:29:07,685 ERROR [org.jboss.as.controller.management-operation] (http-/0.0.0.0:7080-72) JBAS014612: Operation ("read-attribute") failed - address: ([("deployment" => "rhq.ear")]): java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256) [rt.jar:1.8.0_66] at java.util.HashMap.putVal(HashMap.java:630) [rt.jar:1.8.0_66] at java.util.HashMap.put(HashMap.java:611) [rt.jar:1.8.0_66] at java.util.HashSet.add(HashSet.java:219) [rt.jar:1.8.0_66] at org.jboss.as.controller.registry.AbstractModelResource.getChildren(AbstractModelResource.java:115) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:680) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.OperationContextImpl.readResource(OperationContextImpl.java:628) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecuteInternal(ReadAttributeHandler.java:153) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecute(ReadAttributeHandler.java:96) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:249) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:563) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2] at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:283) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA] at org.rhq.common.jbossas.client.controller.DeploymentJBossASClient.getDeploymentPath(DeploymentJBossASClient.java:99) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA] at org.rhq.enterprise.server.core.CoreServer.getEarDeploymentDir(CoreServer.java:187) [rhq-server.jar:4.12.0.JON330GA-redhat-4] at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) [:1.8.0_66] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66] at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66] at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [rt.jar:1.8.0_66] at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) [:1.8.0_66] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66] at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66] at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) [rt.jar:1.8.0_66] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [rt.jar:1.8.0_66] at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [rt.jar:1.8.0_66] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [rt.jar:1.8.0_66] ******************************* Version-Release number of selected component (if applicable): JBoss ON 3.3.4 How reproducible: Always Steps to Reproduce: 1. Make sure that JBoss ON database - rhq_partition_event table contains more then a million events. 2. Attempt to delete partition events by clicking on "Purge All" button in JBoss ON UI -> Administration -> Partition Events page; 3. Actual results: Deletion of partition events fails. Expected results: Partition events are properly deleted. Additional info: "Purge All" - delete of partition events should be done using smaller batches.