Bug 1015560 - Uploading content using remote API/CLI requires many times more heap then file size
Summary: Uploading content using remote API/CLI requires many times more heap then fil...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: CLI
Version: JON 3.2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ER04
: JON 3.2.0
Assignee: Thomas Segismont
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On:
Blocks: 1012435 1017246
TreeView+ depends on / blocked
 
Reported: 2013-10-04 13:51 UTC by Libor Zoubek
Modified: 2015-11-02 00:43 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
: 1017246 (view as bug list)
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Libor Zoubek 2013-10-04 13:51:05 UTC
Description of problem:This bug is very similar to Bug 955363. There are several methods in remote API that still require huge amount of memory when uploading content

BundleManagerRemote.createBundleVersionViaByteArray - for uploading bundles
ResourceFactoryManagerRemote.createPackageBackedResource - for creating resources with content

both methods accept byte[] (representing file content) as argument, this means whole file has to be in memory at least once.


Version-Release number of selected component (if applicable):
JON 3.2.ER2

How reproducible: always


Steps to Reproduce:
1. have a 50MB files (war and bundle)
2. start CLI with default heap size
3a run: 

var file = new java.io.File("/path/to/large/bundle.zip");

var inputStream = new java.io.FileInputStream(file);
var fileLength = file.length();
var fileBytes = java.lang.reflect.Array.newInstance(java.lang.Byte.TYPE, fileLength);
for (numRead=0, offset=0; ((numRead >= 0) && (offset < fileBytes.length)); offset += numRead ) {
   numRead = inputStream.read(fileBytes, offset, fileBytes.length - offset);
}
BundleManager.createBundleVersionViaByteArray(fileBytes);

3b: with large WAR follow https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-Creating_a_new_resource.html

Actual results: In both cases, CLI process failes on OutOfMemory


Expected results: both cases must work, because file size is much smaller than CLI heapsize


Additional info:

Comment 1 Thomas Segismont 2013-10-09 07:45:16 UTC
(In reply to Libor Zoubek from comment #0)
> 3b: with large WAR follow
> https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/
> 2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-
> Creating_a_new_resource.html

This is the up to date version of the doc:
https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/3.1/html-single/Dev_Writing_JON_Command-Line_Scripts/index.html#content-resources

Comment 2 Thomas Segismont 2013-10-09 14:58:43 UTC
Fixed in release/jon3.2.x

commit b531420040505ed63bfeeb68ff3979f4cdb233cd
Author: Thomas Segismont <tsegismo>
Date:   Wed Oct 9 15:44:04 2013 +0200

    Bug 1017246 - Uploading content using remote API/CLI requires many times more heap then file size
    
    Cherry picked from master 7efed3c
    
    Added new methods to create a bundle version or a package backed resource with bytes denoted by a content handle (see ContentManager#createTemporaryContentHandle)
    
    Added method in ScriptUtil to upload a local file and get a content handle
    
    Updated Javascript samples (bundles.js)

Comment 3 Simeon Pinder 2013-10-24 04:09:39 UTC
Moving to ON_QA for testing in the next build.

Comment 4 Libor Zoubek 2013-11-01 08:56:23 UTC
on ER4


Note You need to log in before you can comment on or make changes to this bug.