Bug 879051

Summary: Large files deployed or managed in content system (WAR/EAR/bundle/etc) can cause OutOfMemory failures
Product: [Other] RHQ Project Reporter: Larry O'Leary <loleary>
Component: ContentAssignee: RHQ Project Maintainer <rhq-maint>
Status: NEW --- QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.5CC: hrupp
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 829399 Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 829399    

Description Larry O'Leary 2012-11-21 18:11:07 EST
+++ This bug was initially created as a clone of Bug #829399 +++

Created attachment 589943 [details]
error log

Description of problem:
While trying to deploy a large war (>250 MB) under EAP6 via JON_GUI, it shows that deployment went OK on GUI, while there is a java heap space - OutOfMemory exception in server log.

Version-Release number of selected component (if applicable):
JON - Version: 3.1.0.ER6

How reproducible:
always

Steps to Reproduce:
1. Create child resource under EAP6 as Deployment resource
2. Upload a big war (> 250 MB)
3. Click on Next -> Finish
  
Actual results:
Big war should have been deployed under EAP6

Expected results:
War is not being deployed under EAP and green message about deployment is being shown. (java heap space error apears.)

Additional info:
Please get attached the log.
java heap space params - "-Xms1024M -Xmx1024M -XX:PermSize=256M -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack=true"

--- Additional comment from Armine on 2012-06-06 11:44:36 EDT ---

Hi all,

Please ask me for the large war, haven't yet found the best way/place to keep it.

Regards,
Armine H

--- Additional comment from Heiko W. Rupp on 2012-06-08 08:20:17 EDT ---

Have you tried this with any other plugin too?
What you describe sounds like a out of memory in the server and seems to have 
nothing to do with the As7-plugin.

--- Additional comment from Stefan Negrea on 2012-08-07 18:15:08 EDT ---

Please retest the same scenario but with the AS5 or AS4 plugin. As Heiko commented, the issues seems to be related to the content system and not AS7 plugin in particular.

--- Additional comment from Armine on 2012-08-09 05:39:54 EDT ---

Hi Stefan,

I've checked with EAP5 as well and it's getting java.lang.InterruptedException. Please get attached log.

Regards,
Armine H

--- Additional comment from Armine on 2012-08-09 05:40:38 EDT ---

Created attachment 603219 [details]
error_eap5.log

--- Additional comment from Armine on 2012-08-09 07:20:17 EDT ---

Putting this back to ON_DEV, and later we could change either the title (like EAP6) or close and file another bug.

--- Additional comment from Stefan Negrea on 2012-08-14 13:09:00 EDT ---

This is a general problem related to memory allocation. The errors occur because a large file is stored in memory during the deployment process. While the problem was spotted so far on the server that is not the only place where it can occur. 

The life-cycle of a file:
1) User uploads the file on the server
2) The server stores the file in the database
3) The server sends the file to the agent
4) The agent delivers the file to the managed resource

This process can fail at any point due to running out of memory. Even if the server has enough memory available, that might not be the case for the agent, the managed resource, or even the database itself. The point of failure is unpredictable and it all depends on the configuration and current state of each component of this deployment process.

This a design and implementation problem with the current content system. While there are some better techniques to deal with such large content, the current implementation of the content system uses a memory intensive solution. This is not a regression. As seen so far, the content system works with moderately sized files.

--- Additional comment from Charles Crouch on 2012-08-29 12:07:31 EDT ---

As per 8/20 triage 8/20 pushing to JON312

--- Additional comment from Armine on 2012-10-15 17:23:37 EDT ---

refers to bundle as well.

--- Additional comment from Mike Foley on 2012-10-18 15:45:54 EDT ---

if deployment fails ...for whatever reason the JON UI should convey this to the user ... 

JON should display information in the JON UI (not in the server log ... not in the agent log) that a failure occured.

the actual out-of-memory issue ... that is another issue.

--- Additional comment from Mike Foley on 2012-10-18 15:46:20 EDT ---

if deployment fails ...for whatever reason the JON UI should convey this to the user ... 

JON should display information in the JON UI (not in the server log ... not in the agent log) that a failure occured.

the actual out-of-memory issue ... that is another issue.

--- Additional comment from mark yarborough on 2012-11-20 15:45:28 EST ---

Per triage with loleary, crouch, mfoley: Move to JBoss ON product, set target release JON 3.2, clear priority (will be subject to further triage in JON 3.2 timeframe).