Bug 1297702 - Deletion of partition events in JBoss ON results in OutOfMemoryError when there is a million or more partition events to be deleted
Deletion of partition events in JBoss ON results in OutOfMemoryError when the...
Status: CLOSED ERRATA
Product: JBoss Operations Network
Classification: JBoss
Component: Database (Show other bugs)
JON 3.3.4
Unspecified Unspecified
high Severity high
: DR01
: JON 3.3.6
Assigned To: Michael Burman
Filip Brychta
: Triaged
Depends On: 1297706
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-12 04:32 EST by bkramer
Modified: 2016-07-27 11:32 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 11:32:14 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2120431 None None None 2016-01-12 04:50 EST

  None (edit)
Description bkramer 2016-01-12 04:32:26 EST
Description of problem:

Attempt to delete huge number of partition events (more then 1 million events in the database) fails with error:
*******************************
10:29:07,685 ERROR [org.jboss.as.controller.management-operation] (http-/0.0.0.0:7080-72) JBAS014612: Operation ("read-attribute") failed - address: ([("deployment" => "rhq.ear")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256) [rt.jar:1.8.0_66]
    at java.util.HashMap.putVal(HashMap.java:630) [rt.jar:1.8.0_66]
    at java.util.HashMap.put(HashMap.java:611) [rt.jar:1.8.0_66]
    at java.util.HashSet.add(HashSet.java:219) [rt.jar:1.8.0_66]
    at org.jboss.as.controller.registry.AbstractModelResource.getChildren(AbstractModelResource.java:115) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:680) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.OperationContextImpl.readResource(OperationContextImpl.java:628) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecuteInternal(ReadAttributeHandler.java:153) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecute(ReadAttributeHandler.java:96) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:249) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:563) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
    at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:283) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA]
    at org.rhq.common.jbossas.client.controller.DeploymentJBossASClient.getDeploymentPath(DeploymentJBossASClient.java:99) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA]
    at org.rhq.enterprise.server.core.CoreServer.getEarDeploymentDir(CoreServer.java:187) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
    at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) [:1.8.0_66]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
    at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
    at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [rt.jar:1.8.0_66]
    at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) [:1.8.0_66]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
    at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
    at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) [rt.jar:1.8.0_66]
    at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [rt.jar:1.8.0_66]
    at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [rt.jar:1.8.0_66]
    at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [rt.jar:1.8.0_66]
*******************************

Version-Release number of selected component (if applicable):
JBoss ON 3.3.4

How reproducible:
Always

Steps to Reproduce:
1. Make sure that JBoss ON database - rhq_partition_event table contains more then a million events.
2. Attempt to delete partition events by clicking on "Purge All" button in JBoss ON UI -> Administration -> Partition Events page;
3.

Actual results:
Deletion of partition events fails.

Expected results:
Partition events are properly deleted.

Additional info:
"Purge All" - delete of partition events should be done using smaller batches.
Comment 1 Michael Burman 2016-01-14 15:25:15 EST
Duplicate of BZ 1255196 (fixed in 3.3.5)
Comment 2 Larry O'Leary 2016-01-14 16:05:18 EST
@Michael, I do not think the fix for Bug 1255196 addresses this issue. Looking at the code, I can see we load all events into memory[1] when attempting a purge of all partition events. I would expect that paging should be used or some other method should be used to perform the purge.



[1]: https://github.com/rhq-project/rhq/blob/release/jon3.3.x/modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/cloud/PartitionEventManagerBean.java#L159
Comment 3 Michael Burman 2016-01-15 04:07:26 EST
Right, the customer should use automatic cleanup of partition events (enabled from system settings). For that, the 3.3.5 fixes the out of memory issues as it does not load everything to memory.

At least that seems to be wish in the customer ticket to keep it clean.
Comment 4 dsteigne 2016-01-15 08:10:35 EST
@Michael, is the automatic cleanup of partition events a new system setting?  I do not see that in version 3.3.4
Comment 5 dsteigne 2016-01-15 11:02:36 EST
The Advanced Purge Settings tab is not in JBoss ON 3.3.4 only RHQ 4.13 how can this be added to the Jboss ON release?
Comment 6 Larry O'Leary 2016-01-15 11:29:36 EST
@dsteigne, it cannot. This is not supported in the product and is a community only feature.

This issue will require a fix as described in comment 2.
Comment 7 dsteigne 2016-01-15 11:30:30 EST
@Larry, ok thank you.
Comment 10 Michael Burman 2016-01-26 08:26:47 EST
Fixed in the master:

commit 2a589c3867f931d00845c7f3c303d0d3197b96be
Author: Michael Burman <miburman@redhat.com>
Date:   Tue Jan 26 15:25:55 2016 +0200

    [BZ 1297702] Change purgeAllEvents to use PurgeManager to avoid OutOfMemory issues
Comment 11 Mike McCune 2016-03-28 19:05:20 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 15 Simeon Pinder 2016-06-17 21:11:57 EDT
Moving to ON_QA as available to test with JON 3.3.6 DR01 brew build:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=499890
Comment 17 errata-xmlrpc 2016-07-27 11:32:14 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1519.html

Note You need to log in before you can comment on or make changes to this bug.