Bug 1015560

Summary: Uploading content using remote API/CLI requires many times more heap then file size
Product: [JBoss] JBoss Operations Network Reporter: Libor Zoubek <lzoubek>
Component: CLIAssignee: Thomas Segismont <tsegismo>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: medium Docs Contact:
Priority: unspecified    
Version: JON 3.2CC: myarboro, theute, tsegismo
Target Milestone: ER04   
Target Release: JON 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1017246 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1012435, 1017246    

Description Libor Zoubek 2013-10-04 13:51:05 UTC
Description of problem:This bug is very similar to Bug 955363. There are several methods in remote API that still require huge amount of memory when uploading content

BundleManagerRemote.createBundleVersionViaByteArray - for uploading bundles
ResourceFactoryManagerRemote.createPackageBackedResource - for creating resources with content

both methods accept byte[] (representing file content) as argument, this means whole file has to be in memory at least once.


Version-Release number of selected component (if applicable):
JON 3.2.ER2

How reproducible: always


Steps to Reproduce:
1. have a 50MB files (war and bundle)
2. start CLI with default heap size
3a run: 

var file = new java.io.File("/path/to/large/bundle.zip");

var inputStream = new java.io.FileInputStream(file);
var fileLength = file.length();
var fileBytes = java.lang.reflect.Array.newInstance(java.lang.Byte.TYPE, fileLength);
for (numRead=0, offset=0; ((numRead >= 0) && (offset < fileBytes.length)); offset += numRead ) {
   numRead = inputStream.read(fileBytes, offset, fileBytes.length - offset);
}
BundleManager.createBundleVersionViaByteArray(fileBytes);

3b: with large WAR follow https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-Creating_a_new_resource.html

Actual results: In both cases, CLI process failes on OutOfMemory


Expected results: both cases must work, because file size is much smaller than CLI heapsize


Additional info:

Comment 1 Thomas Segismont 2013-10-09 07:45:16 UTC
(In reply to Libor Zoubek from comment #0)
> 3b: with large WAR follow
> https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/
> 2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-
> Creating_a_new_resource.html

This is the up to date version of the doc:
https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/3.1/html-single/Dev_Writing_JON_Command-Line_Scripts/index.html#content-resources

Comment 2 Thomas Segismont 2013-10-09 14:58:43 UTC
Fixed in release/jon3.2.x

commit b531420040505ed63bfeeb68ff3979f4cdb233cd
Author: Thomas Segismont <tsegismo>
Date:   Wed Oct 9 15:44:04 2013 +0200

    Bug 1017246 - Uploading content using remote API/CLI requires many times more heap then file size
    
    Cherry picked from master 7efed3c
    
    Added new methods to create a bundle version or a package backed resource with bytes denoted by a content handle (see ContentManager#createTemporaryContentHandle)
    
    Added method in ScriptUtil to upload a local file and get a content handle
    
    Updated Javascript samples (bundles.js)

Comment 3 Simeon Pinder 2013-10-24 04:09:39 UTC
Moving to ON_QA for testing in the next build.

Comment 4 Libor Zoubek 2013-11-01 08:56:23 UTC
on ER4