Created attachment 1326356 [details]
key-material.zip

Description of problem:
After upgrade to EAP 6.4.17 the CLI connection with 2-was SSL configured very often fails.

The problem could be related caused by Remoting upgrade from version 3.3.9 to 3.3.10.

https://github.com/jboss-remoting/jboss-remoting/compare/3.3.9.Final...3.3.10.Final


Steps to Reproduce:
* start EAP 6.4.17
* unzip attached key-material.zip into your /tmp (it contains server and client key material - JKS keystores and truststores)
* Run following CLI commands:

jboss-eap-6.4/bin/jboss-cli.sh -c <<EOT
/core-service=management/security-realm=CertificateRealm:add()
/core-service=management/security-realm=CertificateRealm/server-identity=ssl:add(keystore-path=/tmp/server.keystore, keystore-password=123456, alias="cn=server")
/core-service=management/security-realm=CertificateRealm/authentication=truststore:add(keystore-path=/tmp/server.truststore,keystore-password=123456)
/core-service=management/management-interface=native-interface:write-attribute(name=security-realm,value=CertificateRealm)
:reload
EOT

* Store attached jboss-cli.xml into your /tmp (it adds {{<ssl>...</ssl>}} part of the client configuration)
* run loop which calls a JBoss CLI command

for i in $(seq 1 100); do
jboss-eap-6.4/bin/jboss-cli.sh -Djboss.cli.config=/tmp/jboss-cli.xml --connect --timeout=30000 --command="echo Hello $i"
done


Actual results:
The CLI call fails several times with:

Failed to connect to the controller: The controller is not available at localhost:9999: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
	at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:306)
	at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:271)
	at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.modules.Module.run(Module.java:317)
	at org.jboss.modules.Main.main(Main.java:473)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at localhost:9999
	at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:1075)
	at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:903)
	at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:879)
	at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:304)
	... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:149)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:75)
	at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:1053)
	... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:135)
	at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
	at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:208)
	at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:169)
	at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:129)
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:123)
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:263)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:168)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:147)
	... 13 more


Expected results:

All calls in the loop finish with success.