Bug 1015560 - Uploading content using remote API/CLI requires many times more heap then file size
Uploading content using remote API/CLI requires many times more heap then fil...
Status: CLOSED CURRENTRELEASE
Product: JBoss Operations Network
Classification: JBoss
Component: CLI (Show other bugs)
JON 3.2
Unspecified Unspecified
unspecified Severity medium
: ER04
: JON 3.2.0
Assigned To: Thomas Segismont
Mike Foley
:
Depends On:
Blocks: 1012435 1017246
  Show dependency treegraph
 
Reported: 2013-10-04 09:51 EDT by Libor Zoubek
Modified: 2015-11-01 19:43 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1017246 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Libor Zoubek 2013-10-04 09:51:05 EDT
Description of problem:This bug is very similar to Bug 955363. There are several methods in remote API that still require huge amount of memory when uploading content

BundleManagerRemote.createBundleVersionViaByteArray - for uploading bundles
ResourceFactoryManagerRemote.createPackageBackedResource - for creating resources with content

both methods accept byte[] (representing file content) as argument, this means whole file has to be in memory at least once.


Version-Release number of selected component (if applicable):
JON 3.2.ER2

How reproducible: always


Steps to Reproduce:
1. have a 50MB files (war and bundle)
2. start CLI with default heap size
3a run: 

var file = new java.io.File("/path/to/large/bundle.zip");

var inputStream = new java.io.FileInputStream(file);
var fileLength = file.length();
var fileBytes = java.lang.reflect.Array.newInstance(java.lang.Byte.TYPE, fileLength);
for (numRead=0, offset=0; ((numRead >= 0) && (offset < fileBytes.length)); offset += numRead ) {
   numRead = inputStream.read(fileBytes, offset, fileBytes.length - offset);
}
BundleManager.createBundleVersionViaByteArray(fileBytes);

3b: with large WAR follow https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-Creating_a_new_resource.html

Actual results: In both cases, CLI process failes on OutOfMemory


Expected results: both cases must work, because file size is much smaller than CLI heapsize


Additional info:
Comment 1 Thomas Segismont 2013-10-09 03:45:16 EDT
(In reply to Libor Zoubek from comment #0)
> 3b: with large WAR follow
> https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/
> 2.4/html/CLI_Guide/Scripted_Group_Deployments_using_the_CLI_API-
> Creating_a_new_resource.html

This is the up to date version of the doc:
https://access.redhat.com/site/documentation/en-US/JBoss_Operations_Network/3.1/html-single/Dev_Writing_JON_Command-Line_Scripts/index.html#content-resources
Comment 2 Thomas Segismont 2013-10-09 10:58:43 EDT
Fixed in release/jon3.2.x

commit b531420040505ed63bfeeb68ff3979f4cdb233cd
Author: Thomas Segismont <tsegismo@redhat.com>
Date:   Wed Oct 9 15:44:04 2013 +0200

    Bug 1017246 - Uploading content using remote API/CLI requires many times more heap then file size
    
    Cherry picked from master 7efed3c
    
    Added new methods to create a bundle version or a package backed resource with bytes denoted by a content handle (see ContentManager#createTemporaryContentHandle)
    
    Added method in ScriptUtil to upload a local file and get a content handle
    
    Updated Javascript samples (bundles.js)
Comment 3 Simeon Pinder 2013-10-24 00:09:39 EDT
Moving to ON_QA for testing in the next build.
Comment 4 Libor Zoubek 2013-11-01 04:56:23 EDT
on ER4

Note You need to log in before you can comment on or make changes to this bug.