Bug 984306 - Installer fails on app server communication failure
Installer fails on app server communication failure
Status: NEW
Product: RHQ Project
Classification: Other
Component: Installer (Show other bugs)
4.8
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ---
: ---
Assigned To: RHQ Project Maintainer
Mike Foley
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-14 13:04 EDT by Darryl Price
Modified: 2015-02-05 20:19 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Description of how the prerequisites were satisfied (13.99 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-07-14 13:04 EDT, Darryl Price
no flags Details

  None (edit)
Description Darryl Price 2013-07-14 13:04:08 EDT
Created attachment 773385 [details]
Description of how the prerequisites were satisfied

Description of problem:

Having satisfied all the pre-requisites:

[hudson@centos rhq-server-4.8.0]$ bin/rhqctl install --storage-data-root-dir=/experimental/hudson/rhq/rhq-server-4.8.0/cassandradb


12:39:12,110 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
12:39:17,191 INFO  [org.rhq.server.control.command.Install] Preparing to install RHQ storage node.
Starting RHQ Storage Installer ...
12:39:29,891 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
12:39:33,337 INFO  [org.rhq.storage.installer.StorageInstaller] Running RHQ Storage Node installer...
12:39:34,505 INFO  [org.rhq.cassandra.Deployer] Unzipping storage node to /experimental/hudson/rhq/rhq-server-4.8.0/rhq-storage
12:39:43,490 INFO  [org.rhq.cassandra.Deployer] Applying configuration changes to /experimental/hudson/rhq/rhq-server-4.8.0/rhq-storage/conf/cassandra.yaml
12:39:46,132 INFO  [org.rhq.cassandra.Deployer] Applying configuration changes to /experimental/hudson/rhq/rhq-server-4.8.0/rhq-storage/conf/log4j-server.properties
12:39:46,293 INFO  [org.rhq.cassandra.Deployer] Applying configuration changes to /experimental/hudson/rhq/rhq-server-4.8.0/rhq-storage/conf/cassandra-env.sh
12:39:47,261 INFO  [org.rhq.cassandra.Deployer] Updating file permissions in /experimental/hudson/rhq/rhq-server-4.8.0/rhq-storage/bin
12:39:47,346 INFO  [org.rhq.storage.installer.StorageInstaller] Finished installing RHQ Storage Node.
12:39:47,445 INFO  [org.rhq.storage.installer.StorageInstaller] Updating rhq-server.properties...
12:39:48,283 INFO  [org.rhq.storage.installer.StorageInstaller] Starting RHQ Storage Node
12:41:42,239 INFO  [org.rhq.storage.installer.StorageInstaller] RHQ Storage Node is up and running and ready to service client requests
12:41:42,240 INFO  [org.rhq.storage.installer.StorageInstaller] Installation of the storage node has completed successfully.
12:41:42,508 INFO  [org.rhq.server.control.command.Install] The storage node installer has finished with an exit value of 0
12:41:42,529 INFO  [org.rhq.server.control.command.Install] The RHQ Server must be started to complete its upgrade. Starting the RHQ server in preparation of running the server installer...
12:41:43,597 INFO  [org.rhq.server.control.command.Install] Waiting for the RHQ Server to start in preparation of running the server installer for upgrade...
Trying to start the RHQ Server...
RHQ Server                     (pid 19723  ) IS starting
12:42:22,402 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
12:42:31,082 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] The server is preconfigured and ready for auto-install.
12:42:38,528 INFO  [org.xnio] XNIO Version 3.0.7.GA
12:42:39,257 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.7.GA
12:42:40,135 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.14.GA
12:43:08,085 ERROR [org.rhq.enterprise.server.installer.Installer] An error occurred: java.lang.Exception: Cannot obtain client connection to the RHQ app server!!
        at org.rhq.enterprise.server.installer.InstallerServiceImpl.testModelControllerClient(InstallerServiceImpl.java:1034) [rhq-installer-util-4.8.0.jar:4.8.0]
        at org.rhq.enterprise.server.installer.InstallerServiceImpl.preInstall(InstallerServiceImpl.java:197) [rhq-installer-util-4.8.0.jar:4.8.0]
        at org.rhq.enterprise.server.installer.InstallerServiceImpl.test(InstallerServiceImpl.java:122) [rhq-installer-util-4.8.0.jar:4.8.0]
        at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:89) [rhq-installer-util-4.8.0.jar:4.8.0]
        at org.rhq.enterprise.server.installer.Installer.main(Installer.java:56) [rhq-installer-util-4.8.0.jar:4.8.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_35]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_35]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_35]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_35]
        at org.jboss.modules.Module.run(Module.java:262) [jboss-modules.jar:1.2.0.CR1]
        at org.jboss.modules.Main.main(Main.java:329) [jboss-modules.jar:1.2.0.CR1]
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://127.0.0.1:9999. The connection timed out
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:81) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:263) [rhq-jboss-as-dmr-client-4.8.0.jar:4.8.0]
        at org.rhq.common.jbossas.client.controller.CoreJBossASClient.getSystemProperties(CoreJBossASClient.java:103) [rhq-jboss-as-dmr-client-4.8.0.jar:4.8.0]
        at org.rhq.enterprise.server.installer.InstallerServiceImpl.testModelControllerClient(InstallerServiceImpl.java:985) [rhq-installer-util-4.8.0.jar:4.8.0]
        ... 10 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://127.0.0.1:9999. The connection timed out
        at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:130) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:176) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.controller.client.impl.RemotingModelControllerClient.getOrCreateChannel(RemotingModelControllerClient.java:144) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.controller.client.impl.RemotingModelControllerClient$1.getChannel(RemotingModelControllerClient.java:65) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:115) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
        ... 14 more

12:43:08,730 ERROR [org.rhq.server.control.command.Install] An error occurred while starting the RHQ server: Process exited with an error: 2 (Exit value: 2)
12:43:08,777 ERROR [org.rhq.server.control.RHQControl] An error occurred while executing the install command [Cause: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)]
[hudson@centos rhq-server-4.8.0]$


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  Read install documentation
2.  Satisfy all prerequisites to install
3.  run the install with bin/rhqctl install

Actual results:

12:43:08,730 ERROR [org.rhq.server.control.command.Install] An error occurred while starting the RHQ server: Process exited with an error: 2 (Exit value: 2)
12:43:08,777 ERROR [org.rhq.server.control.RHQControl] An error occurred while executing the install command [Cause: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)]
[hudson@centos rhq-server-4.8.0]$


Expected results:

Installation completes cleanly

Additional info:

See attached spreadsheet of prequisites
Comment 1 John Mazzitelli 2013-07-15 15:50:04 EDT
we've seen this before, in odd situations. But I don't think we ever got definitive reasons for it.

This is a very generic error - it is saying that the installer (running on one Java VM) cannot connect to the RHQ Server itself (another Java VM - its just a JBoss AS instance) over the standard port 9999 bound to the loopback device (this is the normal management port of JBossAS).

Is there a firewall somehow preventing connections to the loopback device's 9999 port? Is the server actually able to start (if the server failed to start, the isntaller can't talk to it).

it might be best to start over from scratch - unzip the distro in a clean directory, re-run the install steps, and then, when you get the error, attach to this BZ all the log files in the logs/ directory. Hopefully, something in there will tell us what is going wrong.

Its possible there is a bug, but again, we've had a few peopel report this error message and it usually environmental. But perhaps this could mean we need to do more error-checknig or even have better docs to work around some edge cases?

Note You need to log in before you can comment on or make changes to this bug.