Bug 1175957

Description of problem:
As reported in Bug 1166383, bundle deployment can fail in the event that the connection pool contains a stale or dead socket. For non-SSL connections this is usually not an issue as the error will trigger JBoss Remoting to silently retry the connection and the corresponding request. However, when using SSL, such errors are wrapped in a generic IO exception.

Although the fix for Bug 1166383 should resolve this by using a JBoss Remoting configuration that will allow generic exceptions to be retried, we should include a final retry within JBossRemotingRemoteCommunicator.rawSend in case the remoting configuration has been updated by the user to exclude the necessary options to force the retry. 

This is being classified as a regression due to the same issue being reported in JBoss ON 3.1.2 and being resolved with commit 533a9ad74225fe952a1a553bf1c3588d98f35c06


Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Always

Steps to Reproduce:
The steps from Bug 1166383 can be used. However, please note that the fix for Bug 1166383 should prevent this issue and no failure will occur.

This BZ is only asking that an additional retry be attempted as a last resort in case there are other manifestations of this issue.

Actual results:
ERROR [org.rhq.enterprise.communications.command.client.ClientCommandSenderTask] (http-/0.0.0.0:7080-5) {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.security-token=EhtA72he/fcxoEpuDEQITmL04UgkR4+Jvmaqz9vcYSE3+8D9XOkC4HnvW/uKbSeMi8Y=, rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[schedule], targetInterfaceName=org.rhq.core.clientapi.agent.bundle.BundleAgentService}]]. Cause: org.jboss.remoting.InvocationFailureException:Unable to perform invocation; nested exception is: 
        javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe -> javax.net.ssl.SSLException:Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe -> javax.net.ssl.SSLException:java.net.SocketException: Broken pipe -> java.net.SocketException:Broken pipe. Cause: org.jboss.remoting.InvocationFailureException: Unable to perform invocation; nested exception is: 
        javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pipe

Expected results:
No errors in the event that the connection/socket was stale