Bug 1063364

+++ This bug was initially created as a clone of Bug #1058267 +++

Description of problem:
$Summary

Version-Release number of selected component (if applicable):
rhq-server-4.10.0-SNAPSHOT
4f60fa0d3c7a

How reproducible:
Always

Steps to Reproduce:
1. unzip rhq-server-4.10.0-SNAPSHOT.zip
2. cd rhq-server-4.10.0-SNAPSHOT/bin/
3. ./rhqctl install
4. set jboss.bind.address to 0.0.0.0

Actual results:
RHQ storage is installed correctly but RHQ server installation failed with:
06:46:15,286 INFO  [org.rhq.server.control.command.Install] The RHQ Server must be started to complete its installation. Starting the RHQ server in preparation of running the server installer...
06:46:15,304 INFO  [org.rhq.server.control.command.Install] Waiting for the RHQ Server to start in preparation of running the server installer...
Trying to start the RHQ Server...
RHQ Server                     (pid 6847   ) is ✘ down
Failed to start - make sure the RHQ Server is fully configured properly
06:46:20,850 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
06:46:20,971 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] The server is preconfigured and ready for auto-install.
06:46:21,055 INFO  [org.xnio] XNIO Version 3.0.7.GA
06:46:21,066 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.7.GA
06:46:21,074 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.14.GA
06:46:31,338 ERROR [org.rhq.enterprise.server.installer.Installer] The installer will now exit due to previous errors: java.lang.Exception: Cannot obtain client connection to the RHQ app server!!
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.testModelControllerClient(InstallerServiceImpl.java:1100) [rhq-installer-util-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.preInstall(InstallerServiceImpl.java:217) [rhq-installer-util-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.test(InstallerServiceImpl.java:142) [rhq-installer-util-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:90) [rhq-installer-util-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.Installer.main(Installer.java:57) [rhq-installer-util-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_24]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.6.0_24]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_24]
	at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_24]
	at org.jboss.modules.Module.run(Module.java:262) [jboss-modules.jar:1.2.0.CR1]
	at org.jboss.modules.Main.main(Main.java:329) [jboss-modules.jar:1.2.0.CR1]
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://127.0.0.1:9999. The connection timed out
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:81) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:270) [rhq-jboss-as-dmr-client-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at org.rhq.common.jbossas.client.controller.CoreJBossASClient.getSystemProperties(CoreJBossASClient.java:103) [rhq-jboss-as-dmr-client-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.testModelControllerClient(InstallerServiceImpl.java:1051) [rhq-installer-util-4.10.0-SNAPSHOT.jar:4.10.0-SNAPSHOT]
	... 10 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://127.0.0.1:9999. The connection timed out
	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:130) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:176) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.RemotingModelControllerClient.getOrCreateChannel(RemotingModelControllerClient.java:144) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.RemotingModelControllerClient$1.getChannel(RemotingModelControllerClient.java:65) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:115) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	... 14 more

06:46:31,352 ERROR [org.rhq.server.control.command.Install] An error occurred while starting the RHQ server: Process exited with an error: 2 (Exit value: 2)
RHQ Server                     (pid 6847   ) is ✘ down

Expected results:
Installation is succesfull

--- Additional comment from Filip Brychta on 2014-01-31 04:51:40 EST ---

Update:
This issue is related to JDK1.6. Installation works correctly on JDK1.7

Fails on:
[hudson@last-rhq-server bin]$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (rhel-1.45.1.11.1.el6-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)


Works on:
[hudson@last-rhq-server bin]$ java -version
java version "1.7.0_03-icedtea"
OpenJDK Runtime Environment (rhel-2.1.el6.7-x86_64)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)

--- Additional comment from Filip Brychta on 2014-01-31 07:46:18 EST ---

This is the first build with this issue http://hudson.qa.jboss.com/hudson/view/RHQ/job/rhq-master-gwt-locales/967/

--- Additional comment from Jirka Kremser on 2014-01-31 07:57:27 EST ---

Can be replicated also on HotSpot JVM (1.6.0_24).

--- Additional comment from Jirka Kremser on 2014-01-31 08:11:55 EST ---

The build above (comment 2) contains 86 changes, by using git bisect, it is in the worse case  ceil(log_2(86)) = 7 "build-install-check" cycles. 

It would be nice to have some established tool based on foreman/jenkins to do this automatically.

I am investigating the commits, looking for the culprit..

--- Additional comment from Jirka Kremser on 2014-02-03 08:55:30 EST ---

In commit d6af564d06ce769d1f, the JVM param ("-XX:StringTableSize=1000003") was added that causes this issue. 

Java 6 has apparently problem with this parameter. In Java 7 and higher, the pooled strings are stored on heap, while on Java 6 and lower it is on permgen space.