Not sure why streaming plugins would fail if the SSL comm is setup correctly because its the same comm layer used for everything else.

Can you confirm you went through this and configured everything according to the docs? https://docs.jboss.org/author/display/RHQ/Securing+Communications

I have confirmed it.  I used the same configuration with RHQ 4.11.0 without any problems.

> Not sure why streaming plugins would fail if the SSL comm is setup correctly
> because its the same comm layer used for everything else.
> 
> Can you confirm you went through this and configured everything according to
> the docs? https://docs.jboss.org/author/display/RHQ/Securing+Communications

I am seeing this in a master build as well. Error messages I see below. Not sure what changed recently, but apparently, something did.

2014-08-25 13:42:22,540 INFO  [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.downloading}Downloading the plugin [rhq-snmptrapd-plugin-4.13.0-SNAPSHOT.jar]...
2014-08-25 13:42:22,608 ERROR [main] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotestream]; cmd-in-response=[true]; config=[{rhq.agent-name=mazztower, rhq.externalizable-strategy=AGENT, rhq.security-token=dJlqEMvxiRFGe4acgNTb42BH1EH1Y5f6nE791heeMX5W/QAoK/mVWmHORnF1voQ+pEI=}]; params=[{streamId=1, invocation=NameBasedInvocation[read]}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
2014-08-25 13:42:22,629 ERROR [main] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotestream]; cmd-in-response=[true]; config=[{rhq.agent-name=mazztower, rhq.externalizable-strategy=AGENT, rhq.security-token=dJlqEMvxiRFGe4acgNTb42BH1EH1Y5f6nE791heeMX5W/QAoK/mVWmHORnF1voQ+pEI=}]; params=[{streamId=1, invocation=NameBasedInvocation[close]}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
2014-08-25 13:42:22,630 WARN  [main] (rhq.core.util.stream.StreamUtil)- Streams could not be closed
org.rhq.enterprise.communications.command.client.RemoteIOException
	at org.rhq.enterprise.communications.command.client.RemoteInputStream.sendRequest(RemoteInputStream.java:283)
	at org.rhq.enterprise.communications.command.client.RemoteInputStream.close(RemoteInputStream.java:186)
	at java.io.BufferedInputStream.close(BufferedInputStream.java:472)
	at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:275)
	at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:212)
	at org.rhq.enterprise.agent.PluginUpdate.getPluginArchive(PluginUpdate.java:343)
	at org.rhq.enterprise.agent.PluginUpdate.downloadPluginWithRetries(PluginUpdate.java:282)
	at org.rhq.enterprise.agent.PluginUpdate.updatePlugins(PluginUpdate.java:219)
	at org.rhq.enterprise.agent.AgentMain.startPluginContainer(AgentMain.java:1995)
	at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:729)
	at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:448)
Caused by: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.
	at org.rhq.enterprise.server.remote.RemoteSafeInvocationHandler.invoke(RemoteSafeInvocationHandler.java:125)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
	at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:416)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:235)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:250)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791)
	at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:1453)
	at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:731)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:305)
	at $Proxy172.processRequest(Unknown Source)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:404)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:142)
	at org.rhq.enterprise.communications.servlet.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:70)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:171)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
	at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911)
	at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
2014-08-25 13:42:22,633 ERROR [main] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.download-failure}Failed to download the plugin [snmptrapd]. Cause: java.lang.RuntimeException: Stream data cannot be copied

when agent makes calls like connectAgent or registerAgent, the jboss/remoting handler used is CommandProcessor and the subsystem is RHQ - as expected.

When using remote streaming, the subsystem is null and the handler is RemoteSafeInvocationHandler. I'm not sure if it has always been that way or not, but this is the difference.

See line 967 in org.jboss.remoting.ServerInvoker.invoke(InvocationRequest)

still not sure what the problem is, just jotting down notes as I find them.

(In reply to John Mazzitelli from comment #4)
> when agent makes calls like connectAgent or registerAgent, the
> jboss/remoting handler used is CommandProcessor and the subsystem is RHQ -
> as expected.
> 
> When using remote streaming, the subsystem is null and the handler is
> RemoteSafeInvocationHandler. I'm not sure if it has always been that way or
> not, but this is the difference.

I think this should be ignored; I think that's the way it always has and should be.

i take that back. I just ran 4.11, and the subsystem coming over the wire *is* RHQ (not null), and the handler being used is CommandProcessor.

Something changed in the remote stream handling... very bad change.

seeing odd behavior here:

org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(HttpServletRequest, HttpServletResponse)

content length is the request is 35010 but the byte buffer is only a single byte. The payload is virtually missing. When we get to this line:

 byte[] out = processRequest(servletInvoker, request, totalByteArray, response);

totalByteArray is only a single byte. Its only got one byte and missing 35009 bytes.

OK, not any closer to finding out WHY, but closer to WHERE the problem is.

If I use servlet transport (in the clear comm, over port 7080), it works. If you debug into here:

org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(HttpServletRequest, HttpServletResponse)

you can see reading the ServletInputStream (variable inputStream) retrieves a full payload (in my case, contentLength=35010, and when all the reading is done, totalByteArray has a length of 35010 as expected).

So far so good. But now change the agent config so it talks over sslservlet protocol, over port 7443, and it breaks. In the same code mentioned above, if you debug it, you will see that inputStream fails to get read fully. When I debug this, my contentLength is still 35010 (as was my first test run, so this is good and to be expected) but the first time through reading the inputStream:

int amtRead = inputStream.read(byteBuffer)

results in amtRead=1. The next read results in 0. No more bytes being read. totalByteArray has length of 1. At this point, everything is broke because we did not receive fully the incoming command from the agent. Everything results in null due to empty payload and it just falls over from here.

I have no idea why this works when communication is over http but breaks when going over https.

getting MUCH closer to why (or at least under what conditions) it is happening.

I think it has to do with the length of the message. I started all over, though still going over https, but this time I registered with an agent name that was 10,000 characters long. Thus increasing the size of the registerAgent message. I got the SAME error. So it doesn't have anything specific to do with remoting streaming - it seems that https traffic is getting foobar'ed when the message is large.

Here's what my agent log showed when I tried to register with an agent name of 10,000 "S" characters:

2014-08-26 16:10:41,387 ERROR [RHQ Agent Registration Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=SSS[...]SSS, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[registerAgent], targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService}]]. Cause: org.rhq.core.util.exception.WrappedRemotingException:InvocationRequest did not supply method.. Cause: java.lang.IllegalArgumentException: [Warning] InvocationRequest did not supply method.

So when this happens, and that servlet cannot read the input stream (as the previous comment mentioned), the payload is now missing, we have no incoming command to process, which manifests itself with this "did not supply method" error (because it was null, along with everything else due to the payload failing to be read).

works when the size of the message is about 7k. So somewhere above 7k, definitely over 12k I'm seeing the servlet input stream fail to stream the data.

I just ran a very quick test on JON 3.3 (which has the newer EAP 6.3.GA under the covers) and I did not see an error. But I need to do fuller testing with agent/server using my own keystore/truststore just to validate its working there.

I suspect it is a bug in the underlying Tomcat connector or EAP core code in EAP 6.3.Alpha - I don't think this is anything to do with RHQ per se. Notice the bug only showed up in RHQ 4.12, but not 4.11 (and 4.12 is when we rebased on EAP 6.3.Alpha).

Created attachment 931492 [details]
test war to use (this isn't RHQ's real rhq-remoting.war)

[NOTE: this large comment is here to document the fact that this is definitely NOT an RHQ bug (that is, it is not a bug in code within RHQ itself) - rather, it is a bug in the underlying EAP server that RHQ is running on.]

See attached for a test war to deploy and test to see the bug (if it exists)

rhq-remoting.war is not RHQ's real remoting war, this is just named that way. It just has a TestServlet. To send data to it, just stream data to http://<host>:<port>/rhq-remoting/TestServlet.

Take EAP 6.3.alpha (which is technically jboss-as-dist-7.4.0.Final-redhat-4.zip - this is what RHQ 4.12 and current master build is running on top of), unzip it, copy this attached rhq-remoting.war to the standalone/deployments directory, and edit standalone/configuration/standalone.xml to add an ssl connector that looks like this:

     <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" enable-lookups="false" secure="true" enabled="true" max-connections="200">
         <ssl key-alias="server" password="password" certificate-key-file="/tmp/server.keystore" protocol="TLS" verify-client="false" ca-certificate-file="/tmp/server.truststore" ca-certificate-password="password" keystore-type="JKS" truststore-type="JKS"/>
     </connector>
 
Obviously, you need to create your own self-signed keystore and truststore - just use your own information in the above config to point to your keystore/truststore.

Now run the server via bin/standalone.sh.

Create a small file (just a few bytes is good enough) and a large file (something over, say, 13K bytes). Call them Asmall.txt and Alarge.txt. Now use curl to test. First see that you can stream small data over SSL via:

curl -k --data-binary @/tmp/Asmall.txt https://localhost:8443/rhq-remoting/TestServlet

You will see this output from curl (note: my Asmall.txt file had 13 bytes in it) - this is the response of the servlet - it tells you what the http request's contentLength was (i.e. what we were told to expect to receive) and what we actually read from the servlet input stream:

Total bytes received/read=[13]/[13]

So here you see we were told to expect 13 bytes, and we actually did read 13 bytes. All good.

Now use curl to send a larger stream:

curl -k --data-binary @/tmp/Alarge.txt https://localhost:8443/rhq-remoting/TestServlet

My Alarge.txt is almost 14K bytes. But look - my servlet failed to read any of it:

Total bytes received/read=[13893]/[0]

Now, if I do this entire exercise on EAP 6.3.GA, it all works, both the small and large file can stream successfully. Here's what curl output looks like with the large file on EAP 6.3.GA:

$ curl -k --data-binary @/tmp/Alarge.txt https://localhost:8443/rhq-remoting/TestServlet                                   

Total bytes received/read=[13893]/[13893]

So you can see the servlet was able to read the full amount of bytes that it was expecting. Streaming over non-SSL connector (that is, http://localhost:8080/rhq-remoting.war) with EAP 6.3.GA all works too.

If I go back to EAP 6.3.alpha, and I try to stream over that non-SSL URL, it works with the small data, but with the large data, the servlet CAN read the full amount of data this time, but it still results in an error. Here's what the log message looks like when streaming the large file over non-SSL (http) URL:

09:59:49,727 INFO  [stdout] (http-/127.0.0.1:8080-10) ================TEST SERVLET resultMessage=Total bytes received/read=[13893]/[13893]
09:59:49,728 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rhq-remoting].[TestServlet]] (http-/127.0.0.1:8080-10) JBWEB000236: Servlet.service() for servlet TestServlet threw exception: java.nio.BufferOverflowException                                                
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) [rt.jar:1.7.0_11]                                                                        
Finally, back to the bad EAP 6.3.alpha server when attempting to stream the large data over SSL - here is the full stack:

09:37:03,302 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rhq-remoting].[TestServlet]] (http-/127.0.0.1:8443-44) JBWEB000236: Servlet.service() for servlet TestServlet threw exception: java.nio.BufferOverflowException
	at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) [rt.jar:1.7.0_11]
	at org.apache.coyote.http11.InternalNioOutputBuffer.commit(InternalNioOutputBuffer.java:666) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProcessor.commit(Http11NioProcessor.java:480) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProcessor.action(Http11NioProcessor.java:798) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.Response.action(Response.java:190) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.Response.sendHeaders(Response.java:390) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:352) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:99) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.rhq.enterprise.communications.servlet.TestServlet.processRequest(TestServlet.java:51) [classes:]
	at org.rhq.enterprise.communications.servlet.TestServlet.doPost(TestServlet.java:63) [classes:]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920) [jbossweb-7.4.0.Beta4.jar:7.4.0.Beta4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]

Created attachment 931496 [details]
the TestServlet source code

to be complete, here is the TestServlet source code

This looks related (note it is written against EAP 6.3.alpha as expected):

https://issues.jboss.org/browse/GTNPORTAL-3435

which says is a dup of this:

https://issues.jboss.org/browse/JBWEB-302

THERE IS A SIMPLE WORKAROUND!

From:

https://bugzilla.redhat.com/show_bug.cgi?id=1082560#c2

go in standalone.xml of the EAP 6.3.alpha installation (standalone/configuration/standalone.xml)

find your HTTP connector setting:

<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

See the protocol attribute's value "HTTP/1.1"? Change it to "org.apache.coyote.http11.Http11Protocol" so it looks like:

<connector name="http" protocol="org.apache.coyote.http11.Http11Protocol" scheme="http" socket-binding="http"/>

Do the same thing with your "https" connector:

<connector name="https" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" socket-binding="https" enable-lookups="false" secure="true" enabled="true" max-connections="200">

Then run my replication procedures again as mentioned in the earlier comment #12 and you will see it all works.

For those using RHQ, you'll have to go to <rhq-server-install>/jbossas/standalone/configuration/standalone-full.xml and make those same edits to both http and https connector.

As per comment #15, we'll probably want to look at changing the RHQ installer so it switches the two connector's protocol attribute values to work around this bug.

Note, however, that this isn't needed when RHQ is built on top of EAP 6.3.GA. Just the alpha has the bug. We might want to do some kind of conditional logic - the installer should see what version of EAP it is being installed on top of, and only change the protocol if it is needed.

I have confirmed the workaround on Windows 7 Ultimate SP1 x64.  Thanks for the quick response to this problem.

commit e27ef38cea8b987eb9b435664be01c605f918445 to master. this will now be "fixed" in the next release of RHQ. The installer will now change the http and https connectors so they use the workaround. This will only occur if you install RHQ on top of 7.4.0.Final-redhat-4 (which is EAP 6.3.alpha1).