Bug 1017246 - Uploading content using remote API/CLI requires many times more heap then file size
Summary: Uploading content using remote API/CLI requires many times more heap then fil...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RHQ Project
Classification: Other
Component: CLI
Version: 4.9
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: RHQ 4.10
Assignee: Thomas Segismont
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On: 1015560
Blocks: 1012435
TreeView+ depends on / blocked
 
Reported: 2013-10-09 13:39 UTC by Thomas Segismont
Modified: 2014-04-23 12:30 UTC (History)
4 users (show)

Fixed In Version:
Clone Of: 1015560
Environment:
Last Closed: 2014-04-23 12:30:35 UTC
Embargoed:


Attachments (Terms of Use)

Description Thomas Segismont 2013-10-09 13:39:46 UTC
+++ This bug was initially created as a clone of Bug #1015560 +++

Description of problem:This bug is very similar to Bug 955363. There are several methods in remote API that still require huge amount of memory when uploading content

BundleManagerRemote.createBundleVersionViaByteArray - for uploading bundles
ResourceFactoryManagerRemote.createPackageBackedResource - for creating resources with content

both methods accept byte[] (representing file content) as argument, this means whole file has to be in memory at least once.


Version-Release number of selected component (if applicable):
JON 3.2.ER2

How reproducible: always


Steps to Reproduce:
1. have a 50MB files (war and bundle)
2. start CLI with default heap size
3a run: 

var file = new java.io.File("/path/to/large/bundle.zip");

var inputStream = new java.io.FileInputStream(file);
var fileLength = file.length();
var fileBytes = java.lang.reflect.Array.newInstance(java.lang.Byte.TYPE, fileLength);
for (numRead=0, offset=0; ((numRead >= 0) && (offset < fileBytes.length)); offset += numRead ) {
   numRead = inputStream.read(fileBytes, offset, fileBytes.length - offset);
}
BundleManager.createBundleVersionViaByteArray(fileBytes);

3b: with large WAR follow https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-Creating_a_new_resource.html

Actual results: In both cases, CLI process failes on OutOfMemory


Expected results: both cases must work, because file size is much smaller than CLI heapsize


Additional info:

--- Additional comment from Thomas Segismont on 2013-10-09 09:45:16 CEST ---

(In reply to Libor Zoubek from comment #0)
> 3b: with large WAR follow
> https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/
> 2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-
> Creating_a_new_resource.html

This is the up to date version of the doc:
https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/3.1/html-single/Dev_Writing_JON_Command-Line_Scripts/index.html#content-resources

Comment 1 Thomas Segismont 2013-10-09 13:45:38 UTC
Fixed in master

commit 7efed3c531708c72a628818072af46e95141b812
Author: Thomas Segismont <tsegismo>
Date:   Wed Oct 9 15:44:04 2013 +0200

Added new methods to create a bundle version or a package backed resource with bytes denoted by a content handle (see ContentManager#createTemporaryContentHandle)
    
Added method in ScriptUtil to upload a local file and get a content handle
    
Updated Javascript samples (bundles.js)

Comment 2 Libor Zoubek 2013-10-23 15:38:29 UTC
verified on RHQ 4.10.0 master

Comment 3 Heiko W. Rupp 2014-04-23 12:30:35 UTC
Bulk closing of 4.10 issues.

If an issue is not solved for you, please open a new BZ (or clone the existing one) with a version designator of 4.10.


Note You need to log in before you can comment on or make changes to this bug.