Bug 1133061

Summary: Enabling SSL causes Agent communications to fail during or after plugins download
Product: [Other] RHQ Project Reporter: vandillon
Component: Communications SubsystemAssignee: John Mazzitelli <mazz>
Status: ON_QA --- QA Contact: Mike Foley <mfoley>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.12CC: hrupp, mazz
Target Milestone: GA   
Target Release: RHQ 4.13   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
test war to use (this isn't RHQ's real rhq-remoting.war)
none
the TestServlet source code none

Description vandillon 2014-08-22 15:05:47 UTC
Description of problem: Turning on SSL on 4.12.0 causes communications errors during (agent on Windows platform) or immediately after (agent on Linux platform) plugins download.

Version-Release number of selected component (if applicable): NA

How reproducible: always

Steps to Reproduce:
1. Configure SSL on server
 a) rhq.communications.connector.transport=sslservlet
    rhq.communications.connector.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet
    rhq.server.tomcat.security.client-auth-mode=false
    rhq.server.client.security.server-auth-mode-enabled=false
 b) Add keystore and truststore files to server configuration rhq-server-4.12.0/jbossas/standalone/configuration   

2. Configure SSL on Agent
 a) <entry key="rhq.communications.connector.transport" value="sslsocket" />
    <entry key="rhq.agent.server.transport"  value="sslservlet" />
    <entry key="rhq.agent.server.bind-port"  value="7443" />
 b) Add keystore and truststore files to server configuration rhq-agent/conf   

3. Register agent

Actual results:

1) Linux platform agent:
 a) Agent registers
 b) Plugins download occurs
 c) Exceptions start to occur repeatedly on the server: server.log
    20:35:58,722 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker]] (http-/0.0.0.0:7443-5) ServerInvokerServlet: invokerObjectNameQuery=jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,*
    20:35:58,727 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker]] (http-/0.0.0.0:7443-5) ServerInvokerServlet: Found RHQ remoting servlet: jboss.remoting:service=invoker,transport=sslservlet,host=19.59.214.21,port=7443,rhq.communications.connector.rhqtype=server
    20:36:00,150 INFO  [org.rhq.enterprise.server.core.CoreServerServiceImpl] (http-/0.0.0.0:7443-7) Got agent registration request for new agent: server-agent[19.59.214.21:16163][4.12.0(4905f6e)]
    20:36:00,628 INFO  [org.rhq.enterprise.server.core.CoreServerServiceImpl] (http-/0.0.0.0:7443-9) Agent [server-agent][4.12.0(4905f6e)] would like to connect to this server
    20:36:00,763 INFO  [org.rhq.enterprise.server.core.CoreServerServiceImpl] (http-/0.0.0.0:7443-9) Agent [server-agent] has connected to this server at Thu Aug 21 20:36:00 UTC 2014
    20:36:26,335 ERROR [org.rhq.enterprise.server.remote.RemoteSafeInvocationHandler] (http-/0.0.0.0:7443-25) Failed to invoke remote request: java.lang.IllegalArgumentException: InvocationRequest did not supply method.
        at org.rhq.enterprise.server.remote.RemoteSafeInvocationHandler.invoke(RemoteSafeInvocationHandler.java:93) [rhq-server.jar:4.12.0]
        at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967) [jboss-remoting-2.5.4.SP5.jar:]
        at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:416) [jboss-remoting-2.5.4.SP5.jar:]
     ...
 
 d) no releveant agent.log entries
 e) releveant command-trace.log entries - merge inventory report request sent and no response
    2014-08-21 20:36:25,921 TRACE {send.initiate}==>DiscoveryServerService.mergeInventoryReport|?

2) Windows platform agent:
 a) Agent registers
 b) Plugins download begins and fails, no plugins get downloaded
 c) Exceptions start to occur repeatedly on the server - same exception traces as in 1c above
 d) relevant agent.log exceptions
    2014-08-21 17:00:18,666 ERROR [main] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotestream]; cmd-in-response=[true]; config=[{rhq.agent-name=Beast, rhq.externalizable-strategy=AGENT, rhq.security-token=xLvGxSp58kLF8h91cJ5TbPL0PjnPqjvwGCZXBFuq/DOV9dtSiwbQQJUBdTh4sbeWdIo=}]; params=[{streamId=1, invocation=NameBasedInvocation[read]}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
    2014-08-21 17:00:18,732 ERROR [main] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotestream]; cmd-in-response=[true]; config=[{rhq.agent-name=Beast, rhq.externalizable-strategy=AGENT, rhq.security-token=xLvGxSp58kLF8h91cJ5TbPL0PjnPqjvwGCZXBFuq/DOV9dtSiwbQQJUBdTh4sbeWdIo=}]; params=[{streamId=1, invocation=NameBasedInvocation[close]}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
    2014-08-21 17:00:18,733 WARN  [main] (rhq.core.util.stream.StreamUtil)- Streams could not be closed
    org.rhq.enterprise.communications.command.client.RemoteIOException
        at org.rhq.enterprise.communications.command.client.RemoteInputStream.sendRequest(RemoteInputStream.java:283)
        at org.rhq.enterprise.communications.command.client.RemoteInputStream.close(RemoteInputStream.java:186)
        at java.io.BufferedInputStream.close(BufferedInputStream.java:472)
        at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:275)
        at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:212)
        at org.rhq.enterprise.agent.PluginUpdate.getPluginArchive(PluginUpdate.java:343)
        at org.rhq.enterprise.agent.PluginUpdate.downloadPluginWithRetries(PluginUpdate.java:282)
        at org.rhq.enterprise.agent.PluginUpdate.updatePlugins(PluginUpdate.java:219)
        at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:1934)
        at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:719)
        at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:438)
    Caused by: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
        at org.rhq.enterprise.server.remote.RemoteSafeInvocationHandler.invoke(RemoteSafeInvocationHandler.java:125)
    ...

e) relevant command-trace.log entries - InputStream request failures
    2014-08-21 17:00:18,610 TRACE {send.initiate}==>InputStream.read.1|?
    2014-08-21 17:00:18,666 TRACE {send.complete}=>>InputStream.read.1|?|org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.  
  
Expected results: Successful completion of registration cycle

Additional info:
 Server platform: Ubuntu 12.04 LTS x64
 Agent platforms: Ubuntu 12.04 LTS x64, Windows 7 Ultimate SP1 x64

Comment 1 John Mazzitelli 2014-08-22 21:10:13 UTC
Not sure why streaming plugins would fail if the SSL comm is setup correctly because its the same comm layer used for everything else.

Can you confirm you went through this and configured everything according to the docs? https://docs.jboss.org/author/display/RHQ/Securing+Communications

Comment 2 vandillon 2014-08-22 22:18:12 UTC
I have confirmed it.  I used the same configuration with RHQ 4.11.0 without any problems.

> Not sure why streaming plugins would fail if the SSL comm is setup correctly
> because its the same comm layer used for everything else.
> 
> Can you confirm you went through this and configured everything according to
> the docs? https://docs.jboss.org/author/display/RHQ/Securing+Communications

Comment 3 John Mazzitelli 2014-08-25 17:51:28 UTC
I am seeing this in a master build as well. Error messages I see below. Not sure what changed recently, but apparently, something did.

2014-08-25 13:42:22,540 INFO  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.downloading}Downloading the plugin [rhq-snmptrapd-plugin-4.13.0-SNAPSHOT.jar]...
2014-08-25 13:42:22,608 ERROR [main] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotestream]; cmd-in-response=[true]; config=[{rhq.agent-name=mazztower, rhq.externalizable-strategy=AGENT, rhq.security-token=dJlqEMvxiRFGe4acgNTb42BH1EH1Y5f6nE791heeMX5W/QAoK/mVWmHORnF1voQ+pEI=}]; params=[{streamId=1, invocation=NameBasedInvocation[read]}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
2014-08-25 13:42:22,629 ERROR [main] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotestream]; cmd-in-response=[true]; config=[{rhq.agent-name=mazztower, rhq.externalizable-strategy=AGENT, rhq.security-token=dJlqEMvxiRFGe4acgNTb42BH1EH1Y5f6nE791heeMX5W/QAoK/mVWmHORnF1voQ+pEI=}]; params=[{streamId=1, invocation=NameBasedInvocation[close]}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
2014-08-25 13:42:22,630 WARN  [main] (rhq.core.util.stream.StreamUtil)- Streams could not be closed
org.rhq.enterprise.communications.command.client.RemoteIOException
	at org.rhq.enterprise.communications.command.client.RemoteInputStream.sendRequest(RemoteInputStream.java:283)
	at org.rhq.enterprise.communications.command.client.RemoteInputStream.close(RemoteInputStream.java:186)
	at java.io.BufferedInputStream.close(BufferedInputStream.java:472)
	at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:275)
	at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:212)
	at org.rhq.enterprise.agent.PluginUpdate.getPluginArchive(PluginUpdate.java:343)
	at org.rhq.enterprise.agent.PluginUpdate.downloadPluginWithRetries(PluginUpdate.java:282)
	at org.rhq.enterprise.agent.PluginUpdate.updatePlugins(PluginUpdate.java:219)
	at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:1995)
	at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:729)
	at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:448)
Caused by: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
	at org.rhq.enterprise.server.remote.RemoteSafeInvocationHandler.invoke(RemoteSafeInvocationHandler.java:125)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
	at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:416)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791)
	at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:1453)
	at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:731)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:305)
	at $Proxy172.processRequest(Unknown Source)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:404)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:142)
	at org.rhq.enterprise.communications.servlet.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:70)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:171)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
	at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911)
	at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
2014-08-25 13:42:22,633 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [snmptrapd]. Cause: java.lang.RuntimeException: Stream data cannot be copied

Comment 4 John Mazzitelli 2014-08-26 17:08:56 UTC
when agent makes calls like connectAgent or registerAgent, the jboss/remoting handler used is CommandProcessor and the subsystem is RHQ - as expected.

When using remote streaming, the subsystem is null and the handler is RemoteSafeInvocationHandler. I'm not sure if it has always been that way or not, but this is the difference.

See line 967 in org.jboss.remoting.ServerInvoker.invoke(InvocationRequest)

still not sure what the problem is, just jotting down notes as I find them.

Comment 5 John Mazzitelli 2014-08-26 17:19:02 UTC
(In reply to John Mazzitelli from comment #4)
> when agent makes calls like connectAgent or registerAgent, the
> jboss/remoting handler used is CommandProcessor and the subsystem is RHQ -
> as expected.
> 
> When using remote streaming, the subsystem is null and the handler is
> RemoteSafeInvocationHandler. I'm not sure if it has always been that way or
> not, but this is the difference.

I think this should be ignored; I think that's the way it always has and should be.

Comment 6 John Mazzitelli 2014-08-26 17:51:22 UTC
i take that back. I just ran 4.11, and the subsystem coming over the wire *is* RHQ (not null), and the handler being used is CommandProcessor.

Something changed in the remote stream handling... very bad change.

Comment 7 John Mazzitelli 2014-08-26 19:02:46 UTC
seeing odd behavior here:

org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(HttpServletRequest, HttpServletResponse)

content length is the request is 35010 but the byte buffer is only a single byte. The payload is virtually missing. When we get to this line:

 byte[] out = processRequest(servletInvoker, request, totalByteArray, response);

totalByteArray is only a single byte. Its only got one byte and missing 35009 bytes.

Comment 8 John Mazzitelli 2014-08-26 19:40:26 UTC
OK, not any closer to finding out WHY, but closer to WHERE the problem is.

If I use servlet transport (in the clear comm, over port 7080), it works. If you debug into here:

org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(HttpServletRequest, HttpServletResponse)

you can see reading the ServletInputStream (variable inputStream) retrieves a full payload (in my case, contentLength=35010, and when all the reading is done, totalByteArray has a length of 35010 as expected).

So far so good. But now change the agent config so it talks over sslservlet protocol, over port 7443, and it breaks. In the same code mentioned above, if you debug it, you will see that inputStream fails to get read fully. When I debug this, my contentLength is still 35010 (as was my first test run, so this is good and to be expected) but the first time through reading the inputStream:

int amtRead = inputStream.read(byteBuffer)

results in amtRead=1. The next read results in 0. No more bytes being read. totalByteArray has length of 1. At this point, everything is broke because we did not receive fully the incoming command from the agent. Everything results in null due to empty payload and it just falls over from here.

I have no idea why this works when communication is over http but breaks when going over https.

Comment 9 John Mazzitelli 2014-08-26 20:16:42 UTC
getting MUCH closer to why (or at least under what conditions) it is happening.

I think it has to do with the length of the message. I started all over, though still going over https, but this time I registered with an agent name that was 10,000 characters long. Thus increasing the size of the registerAgent message. I got the SAME error. So it doesn't have anything specific to do with remoting streaming - it seems that https traffic is getting foobar'ed when the message is large.

Here's what my agent log showed when I tried to register with an agent name of 10,000 "S" characters:

2014-08-26 16:10:41,387 ERROR [RHQ Agent Registration Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=SSS[...]SSS, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[registerAgent], targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.

So when this happens, and that servlet cannot read the input stream (as the previous comment mentioned), the payload is now missing, we have no incoming command to process, which manifests itself with this "did not supply method" error (because it was null, along with everything else due to the payload failing to be read).

Comment 10 John Mazzitelli 2014-08-26 20:26:57 UTC
works when the size of the message is about 7k. So somewhere above 7k, definitely over 12k I'm seeing the servlet input stream fail to stream the data.

Comment 11 John Mazzitelli 2014-08-26 21:17:49 UTC
I just ran a very quick test on JON 3.3 (which has the newer EAP 6.3.GA under the covers) and I did not see an error. But I need to do fuller testing with agent/server using my own keystore/truststore just to validate its working there.

I suspect it is a bug in the underlying Tomcat connector or EAP core code in EAP 6.3.Alpha - I don't think this is anything to do with RHQ per se. Notice the bug only showed up in RHQ 4.12, but not 4.11 (and 4.12 is when we rebased on EAP 6.3.Alpha).

Comment 12 John Mazzitelli 2014-08-27 14:08:35 UTC
Created attachment 931492 [details]
test war to use (this isn't RHQ's real rhq-remoting.war)

[NOTE: this large comment is here to document the fact that this is definitely NOT an RHQ bug (that is, it is not a bug in code within RHQ itself) - rather, it is a bug in the underlying EAP server that RHQ is running on.]

See attached for a test war to deploy and test to see the bug (if it exists)

rhq-remoting.war is not RHQ's real remoting war, this is just named that way. It just has a TestServlet. To send data to it, just stream data to http://<host>:<port>/rhq-remoting/TestServlet.

Take EAP 6.3.alpha (which is technically jboss-as-dist-7.4.0.Final-redhat-4.zip - this is what RHQ 4.12 and current master build is running on top of), unzip it, copy this attached rhq-remoting.war to the standalone/deployments directory, and edit standalone/configuration/standalone.xml to add an ssl connector that looks like this:

     <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" enable-lookups="false" secure="true" enabled="true" max-connections="200">
         <ssl key-alias="server" password="password" certificate-key-file="/tmp/server.keystore" protocol="TLS" verify-client="false" ca-certificate-file="/tmp/server.truststore" ca-certificate-password="password" keystore-type="JKS" truststore-type="JKS"/>
     </connector>
 
Obviously, you need to create your own self-signed keystore and truststore - just use your own information in the above config to point to your keystore/truststore.

Now run the server via bin/standalone.sh.

Create a small file (just a few bytes is good enough) and a large file (something over, say, 13K bytes). Call them Asmall.txt and Alarge.txt. Now use curl to test. First see that you can stream small data over SSL via:

curl -k --data-binary @/tmp/Asmall.txt https://localhost:8443/rhq-remoting/TestServlet

You will see this output from curl (note: my Asmall.txt file had 13 bytes in it) - this is the response of the servlet - it tells you what the http request's contentLength was (i.e. what we were told to expect to receive) and what we actually read from the servlet input stream:

Total bytes received/read=[13]/[13]

So here you see we were told to expect 13 bytes, and we actually did read 13 bytes. All good.

Now use curl to send a larger stream:

curl -k --data-binary @/tmp/Alarge.txt https://localhost:8443/rhq-remoting/TestServlet

My Alarge.txt is almost 14K bytes. But look - my servlet failed to read any of it:

Total bytes received/read=[13893]/[0]

Now, if I do this entire exercise on EAP 6.3.GA, it all works, both the small and large file can stream successfully. Here's what curl output looks like with the large file on EAP 6.3.GA:

$ curl -k --data-binary @/tmp/Alarge.txt https://localhost:8443/rhq-remoting/TestServlet                                   

Total bytes received/read=[13893]/[13893]

So you can see the servlet was able to read the full amount of bytes that it was expecting. Streaming over non-SSL connector (that is, http://localhost:8080/rhq-remoting.war) with EAP 6.3.GA all works too.

If I go back to EAP 6.3.alpha, and I try to stream over that non-SSL URL, it works with the small data, but with the large data, the servlet CAN read the full amount of data this time, but it still results in an error. Here's what the log message looks like when streaming the large file over non-SSL (http) URL:

09:59:49,727 INFO  [stdout] (http-/127.0.0.1:8080-10) ================TEST SERVLET resultMessage=Total bytes received/read=[13893]/[13893]
09:59:49,728 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rhq-remoting].[TestServlet]] (http-/127.0.0.1:8080-10) JBWEB000236: Servlet.service() for servlet TestServlet threw exception: java.nio.BufferOverflowException                                                
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) [rt.jar:1.7.0_11]                                                                        
Finally, back to the bad EAP 6.3.alpha server when attempting to stream the large data over SSL - here is the full stack:

09:37:03,302 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rhq-remoting].[TestServlet]] (http-/127.0.0.1:8443-44) JBWEB000236: Servlet.service() for servlet TestServlet threw exception: java.nio.BufferOverflowException
	at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) [rt.jar:1.7.0_11]
	at org.apache.coyote.http11.InternalNioOutputBuffer.commit(InternalNioOutputBuffer.java:666) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProcessor.commit(Http11NioProcessor.java:480) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProcessor.action(Http11NioProcessor.java:798) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.Response.action(Response.java:190) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.Response.sendHeaders(Response.java:390) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:352) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:99) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.rhq.enterprise.communications.servlet.TestServlet.processRequest(TestServlet.java:51) [classes:]
	at org.rhq.enterprise.communications.servlet.TestServlet.doPost(TestServlet.java:63) [classes:]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]

Comment 13 John Mazzitelli 2014-08-27 14:14:54 UTC
Created attachment 931496 [details]
the TestServlet source code

to be complete, here is the TestServlet source code

Comment 14 John Mazzitelli 2014-08-27 14:21:02 UTC
This looks related (note it is written against EAP 6.3.alpha as expected):

https://issues.jboss.org/browse/GTNPORTAL-3435

which says is a dup of this:

https://issues.jboss.org/browse/JBWEB-302

Comment 15 John Mazzitelli 2014-08-27 14:37:43 UTC
THERE IS A SIMPLE WORKAROUND!

From:

https://bugzilla.redhat.com/show_bug.cgi?id=1082560#c2

go in standalone.xml of the EAP 6.3.alpha installation (standalone/configuration/standalone.xml)

find your HTTP connector setting:

<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

See the protocol attribute's value "HTTP/1.1"? Change it to "org.apache.coyote.http11.Http11Protocol" so it looks like:

<connector name="http" protocol="org.apache.coyote.http11.Http11Protocol" scheme="http" socket-binding="http"/>

Do the same thing with your "https" connector:

<connector name="https" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" socket-binding="https" enable-lookups="false" secure="true" enabled="true" max-connections="200">

Then run my replication procedures again as mentioned in the earlier comment #12 and you will see it all works.

For those using RHQ, you'll have to go to <rhq-server-install>/jbossas/standalone/configuration/standalone-full.xml and make those same edits to both http and https connector.

Comment 16 John Mazzitelli 2014-08-27 14:40:45 UTC
As per comment #15, we'll probably want to look at changing the RHQ installer so it switches the two connector's protocol attribute values to work around this bug.

Note, however, that this isn't needed when RHQ is built on top of EAP 6.3.GA. Just the alpha has the bug. We might want to do some kind of conditional logic - the installer should see what version of EAP it is being installed on top of, and only change the protocol if it is needed.

Comment 17 vandillon 2014-08-27 15:40:28 UTC
I have confirmed the workaround on Windows 7 Ultimate SP1 x64.  Thanks for the quick response to this problem.

Comment 18 John Mazzitelli 2014-08-27 20:00:52 UTC
commit e27ef38cea8b987eb9b435664be01c605f918445 to master. this will now be "fixed" in the next release of RHQ. The installer will now change the http and https connectors so they use the workaround. This will only occur if you install RHQ on top of 7.4.0.Final-redhat-4 (which is EAP 6.3.alpha1).