Bug 955363 - Uploading of file content using remote API/CLI requires many times more heap then file size
Summary: Uploading of file content using remote API/CLI requires many times more heap ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: CLI
Version: JON 3.1.2
Hardware: All
OS: All
unspecified
low
Target Milestone: ER02
: JON 3.2.0
Assignee: Thomas Segismont
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-04-22 21:13 UTC by Larry O'Leary
Modified: 2018-12-01 16:06 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2014-01-02 20:37:15 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 335353 0 None None None Never

Description Larry O'Leary 2013-04-22 21:13:02 UTC
Description of problem:
When attempting to upload a WAR that is ~60MB using the CLI, it fails with the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2798)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
	at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1857)
	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1766)
	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
	at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(JavaSerializationManager.java:120)
	at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(JavaSerializationManager.java:95)
	at org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(SerializableMarshaller.java:120)
	at org.jboss.remoting.marshal.http.HTTPMarshaller.write(HTTPMarshaller.java:73)
	at org.jboss.remoting.transport.http.HTTPClientInvoker.useHttpURLConnection(HTTPClientInvoker.java:279)
	at org.jboss.remoting.transport.http.HTTPClientInvoker.transport(HTTPClientInvoker.java:137)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
	at org.jboss.remoting.Client.invoke(Client.java:1634)
	at org.jboss.remoting.Client.invoke(Client.java:548)
	at org.jboss.remoting.Client.invoke(Client.java:536)
	at org.rhq.enterprise.clientapi.RemoteClientProxy.doInvoke(RemoteClientProxy.java:85)
	at org.rhq.bindings.client.AbstractRhqFacadeProxy.invoke(AbstractRhqFacadeProxy.java:87)
	at org.rhq.enterprise.clientapi.RemoteClientProxy.invoke(RemoteClientProxy.java:69)
	at $Proxy14.createPackageVersionWithDisplayVersion(Unknown Source)
	at org.rhq.bindings.client.ResourceClientProxy$ClientProxyMethodHandler.updateBackingContent(ResourceClientProxy.java:545)
	at org.rhq.bindings.client.ResourceClientProxy$ClientProxyMethodHandler.updateBackingContent(ResourceClientProxy.java:519)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.rhq.bindings.client.ResourceClientProxy$ClientProxyMethodHandler.invoke(ResourceClientProxy.java:602)
	at org.rhq.bindings.client.ResourceClientProxy_$$_javassist_0.updateBackingContent(ResourceClientProxy_$$_javassist_0.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


Version-Release number of selected component (if applicable):
4.4.0.JON312GA

How reproducible:
Always

Steps to Reproduce:
1.  Install JBoss ON remote API and CLI (rhq-remoting-cli).
2.  Copy example varia/threaddump.war from JBoss EAP 5 to tmp:
        
        cp -a "${JBOSS_HOME}/docs/examples/varia/threaddump.war" /tmp

3.  Create a file with random contents to create WAR bloat to equal ~60MB:

        dd if=/dev/urandom of=/tmp/random-junk.file bs=1024K count=60

4.  Add random-junk.file to jboss-as-helloworld.war:

        cd /tmp
        zip threaddump.war random-junk.file

5.  Create JBoss ON CLI test script:

        cat >/tmp/upload-large-war-test.js <<EOF
var criteria = new ResourceCriteria();
var resource = new Resource();
criteria.addFilterName("threaddump.war");
criteria.addFilterResourceTypeName('Web Application (WAR)');
criteria.addFilterPluginName('JBossAS');
resource = ResourceManager.findResourcesByCriteria(criteria).get(0);
var managed_resource = ProxyFactory.getResource(resource.id);
while (managed_resource.getBackingContent() == null) {
    // We have to do this because it take some time for the backing content to actually show up.
    java.lang.Thread.currentThread().sleep(5000);
    managed_resource = ProxyFactory.getResource(resource.id);
    // We seep again because it seems that the proxy factory may not have fully constructed the object in time
    java.lang.Thread.currentThread().sleep(1000);
}
managed_resource.updateBackingContent("/tmp/threaddump.war");
EOF

6.  Copy threaddump.war to 'all' deploy directory:

        cp -a /tmp/threaddump.war ${JBOSS_HOME}/server/all/deploy

7.  Start EAP 5.2 'all' server.
8.  Start JBoss ON system.
9.  Import EAP server into inventory.
10. Execute CLI test script:
        
        ./rhq-cli.sh -u rhqadmin -p rhqadmin -s localhost -t 7080 -f /tmp/upload-large-war-test.js

Wait sever minutes as it will take the conent a while to make its way into the database.  

Actual results:
Execution will fail with the following error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2798)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111)
	at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)


Expected results:
Execution should not fail and content should be updated.

Additional info:

Comment 3 Thomas Segismont 2013-09-10 15:54:07 UTC
Fixed in master

commit 0b99feed5411fe1b3437f9e64953c4818a5a4289
Author: Thomas Segismont <tsegismo>
Date:   Tue Sep 10 10:15:52 2013 +0200

The CLI uses JBoss Remoting to communicate with the server. JBoss Remoting relies on JDK's HttpUrlConnection when configured for the HTTP transport. But HttpUrlConnection copies the full payload of the HTTP request in memory before sending it to the server. Therefore, when you send a huge file, you can easily reach the maximum heap.

Another problem was that the CLI itself was loading the file in memory as well, just to compute the file checksum. 

Updated ResourceClientProxy to compute checksum on FileInputStream instead of file bytes.
Created new methods in ContentManager to upload content in fragments, updated ResourceClientProxy accordingly.

Comment 4 Libor Zoubek 2013-10-04 13:38:57 UTC
verified on JON 3.2.0.ER2


Note You need to log in before you can comment on or make changes to this bug.