Bug 971377 - Installation via rhqctl install fails with "The server failed to start up properly"
Installation via rhqctl install fails with "The server failed to start up pro...
Status: ON_DEV
Product: RHQ Project
Classification: Other
Component: Installer (Show other bugs)
4.8
Unspecified Unspecified
unspecified Severity medium (vote)
: ---
: ---
Assigned To: RHQ Project Maintainer
Mike Foley
:
Depends On: 950422
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-06 07:31 EDT by Filip Brychta
Modified: 2015-11-01 19:45 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rhq server log (309.94 KB, text/x-log)
2013-06-06 07:31 EDT, Filip Brychta
no flags Details

  None (edit)
Description Filip Brychta 2013-06-06 07:31:12 EDT
Created attachment 757628 [details]
rhq server log

Description of problem:
I'm trying to install the latest rhq master build via 'rhqctl install' but the installation process (i can see it via 'ps -ef | grep java') hangs on message: 

"06:45:10,662 INFO  [org.rhq.enterprise.server.installer.Installer] Installation is complete. The server should be ready shortly." 

This problem is probably caused by exception in rhq server log:

The server failed to start up properly: javax.ejb.EJBException: java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] 

but the relevant property in rhq-server.properties IS set correctly:

rhq.cassandra.seeds=fbry-test2.bc.jonqe.lab.eng.bos.redhat.com|7299|9142

I can see the error message from bz 950422 during the installation. When i kill the installation process and restart the rhq server: 
./rhq-server.sh stop
./rhq-server.sh start

the installation finishes successfully.

So here is a possible cause:
1- installation process updates rhq-server.properties (rhq.cassandra.seeds)
2- exception from bz 950422 prevents the rhq server from reloading its updated configuration (which already contains correct rhq.cassandra.seeds)
3- rhq server uses the old configuration with empty rhq.cassandra.seeds property
4- exception java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: []  is thrown during the installation to the rhq server log.



Version-Release number of selected component (if applicable):
Version: 4.8.0-SNAPSHOT
Build Number: 9c0489c

How reproducible:
3 out of 3

Steps to Reproduce:
1. have a clean environment
2. create /var/lib/rhq with relevant permissions for non-root user
3. unzip rhq-server-4.8.0-SNAPSHOT.zip
4. cd rhq-server-4.8.0-SNAPSHOT/bin/
5. chmod +x rhqctl
6. ./rhq-server.sh start (this step is a workaround for bz 967525)
7. ./rhqctl install

Actual results:
installation process (i can see it via 'ps -ef | grep java') hangs on message: 

"06:45:10,662 INFO  [org.rhq.enterprise.server.installer.Installer] Installation is complete. The server should be ready shortly.

and server installation fails with exception in rhq server log:

The server failed to start up properly: javax.ejb.EJBException: java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] 

Expected results:
Installation is successful

Additional info:
complete rhq server log attached

installer exception:

06:43:47,884 WARN  [org.rhq.enterprise.server.installer.InstallerServiceImpl] reloadConfiguration failed - restart the server to complete the installation: java.io.IOException: java.util.concurrent.ExecutionException: Operation failed
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:81) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:263) [rhq-jboss-as-dmr-client-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at org.rhq.common.jbossas.client.controller.CoreJBossASClient.reload(CoreJBossASClient.java:373) [rhq-jboss-as-dmr-client-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at org.rhq.common.jbossas.client.controller.CoreJBossASClient.reload(CoreJBossASClient.java:354) [rhq-jboss-as-dmr-client-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.reloadConfiguration(InstallerServiceImpl.java:1145) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:303) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:115) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.Installer.main(Installer.java:56) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_24]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.6.0_24]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_24]
	at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_24]
	at org.jboss.modules.Module.run(Module.java:262) [jboss-modules.jar:1.2.0.CR1]
	at org.jboss.modules.Main.main(Main.java:329) [jboss-modules.jar:1.2.0.CR1]
Caused by: java.util.concurrent.ExecutionException: Operation failed
	at org.jboss.threads.AsyncFutureTask.operationFailed(AsyncFutureTask.java:74)
	at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:268)
	at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:100) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	... 14 more
Caused by: java.io.IOException: Channel closed
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleChannelClosed(AbstractMessageHandler.java:381) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.RemotingModelControllerClient$2.handleClose(RemotingModelControllerClient.java:133) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.as.controller.client.impl.RemotingModelControllerClient$2.handleClose(RemotingModelControllerClient.java:130) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:515)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:153)
	at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleEnd(ManagementChannelReceiver.java:129) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
	at org.jboss.remoting3.remote.RemoteConnectionChannel$2.run(RemoteConnectionChannel.java:277)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24]
	at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]
Comment 1 John Sanda 2013-06-06 12:12:06 EDT
The exception here is coming from the server installer. There was a regression in which we were using the wrong credentials to login into Cassandra and create the schema. The fix for this is available as of build 2292 of the rhq-master Jenkins job.

Moving this to ON_QA since the fix is available.
Comment 2 Filip Brychta 2013-06-07 08:29:19 EDT
Update:
Version: 4.8.0-SNAPSHOT
Build Number: bf44b14

- Installer exception (Caused by: java.io.IOException: Channel closed) is fixed.
- Installer process doesn't hang any more.
- Exception from rhq server log 

The server failed to start up properly: javax.ejb.EJBException: java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] 

is still there and server is still not accessible from browser.

Adding a workaround again (it is a bit hidden in the original description):
./rhq-server.sh stop
./rhq-server.sh start
Comment 3 Filip Brychta 2013-06-11 09:44:31 EDT
Lowering severity. This is no longer blocking issue for our automation.

Note You need to log in before you can comment on or make changes to this bug.