Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1491965

Summary: [GSS](6.4.z) CLI with 2-way SSL often hangs/times out
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Josef Cacek <jcacek>
Component: Remoting, CLIAssignee: Panagiotis Sotiropoulos <psotirop>
Status: CLOSED CURRENTRELEASE QA Contact: Jiří Bílek <jbilek>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4.17CC: bmaxwell, cdewolf, david.lloyd, dboeren, heiner.tittelbach, jbaesner, pjurak, pragshar, psotirop, schaudha, thofman, tmiyargi
Target Milestone: CR1Keywords: Regression
Target Release: EAP 6.4.18   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-06 18:31:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1463709, 1497357, 1498189    
Attachments:
Description Flags
key-material.zip
none
jboss-cli.xml
none
jboss-cli-thread-dumps.zip
none
reproducer-overlay.zip none

Description Josef Cacek 2017-09-15 07:30:27 UTC
Created attachment 1326356 [details]
key-material.zip

Description of problem:
After upgrade to EAP 6.4.17 the CLI connection with 2-was SSL configured very often fails.

The problem could be related caused by Remoting upgrade from version 3.3.9 to 3.3.10.

https://github.com/jboss-remoting/jboss-remoting/compare/3.3.9.Final...3.3.10.Final


Steps to Reproduce:
* start EAP 6.4.17
* unzip attached key-material.zip into your /tmp (it contains server and client key material - JKS keystores and truststores)
* Run following CLI commands:

jboss-eap-6.4/bin/jboss-cli.sh -c <<EOT
/core-service=management/security-realm=CertificateRealm:add()
/core-service=management/security-realm=CertificateRealm/server-identity=ssl:add(keystore-path=/tmp/server.keystore, keystore-password=123456, alias="cn=server")
/core-service=management/security-realm=CertificateRealm/authentication=truststore:add(keystore-path=/tmp/server.truststore,keystore-password=123456)
/core-service=management/management-interface=native-interface:write-attribute(name=security-realm,value=CertificateRealm)
:reload
EOT

* Store attached jboss-cli.xml into your /tmp (it adds {{<ssl>...</ssl>}} part of the client configuration)
* run loop which calls a JBoss CLI command

for i in $(seq 1 100); do
jboss-eap-6.4/bin/jboss-cli.sh -Djboss.cli.config=/tmp/jboss-cli.xml --connect --timeout=30000 --command="echo Hello $i"
done


Actual results:
The CLI call fails several times with:

Failed to connect to the controller: The controller is not available at localhost:9999: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
	at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:306)
	at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:271)
	at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.modules.Module.run(Module.java:317)
	at org.jboss.modules.Main.main(Main.java:473)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at localhost:9999
	at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:1075)
	at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:903)
	at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:879)
	at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:304)
	... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:149)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:75)
	at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:1053)
	... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:135)
	at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
	at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:208)
	at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:169)
	at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:129)
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:123)
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:263)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:168)
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:147)
	... 13 more


Expected results:

All calls in the loop finish with success.

Comment 1 Josef Cacek 2017-09-15 07:31:06 UTC
Created attachment 1326357 [details]
jboss-cli.xml

Comment 6 Brad Maxwell 2017-10-02 19:25:14 UTC
Created attachment 1333396 [details]
jboss-cli-thread-dumps.zip

Comment 7 Brad Maxwell 2017-10-02 19:25:44 UTC
Created attachment 1333397 [details]
reproducer-overlay.zip

Comment 27 Jiří Bílek 2017-10-18 14:14:38 UTC
Verified with EAP 6.4.18.CP.CR1

Comment 28 Petr Penicka 2017-12-06 18:31:05 UTC
Released on Nov 14 2017 as part of EAP 6.4.18.