Created attachment 757628 [details] rhq server log Description of problem: I'm trying to install the latest rhq master build via 'rhqctl install' but the installation process (i can see it via 'ps -ef | grep java') hangs on message: "06:45:10,662 INFO [org.rhq.enterprise.server.installer.Installer] Installation is complete. The server should be ready shortly." This problem is probably caused by exception in rhq server log: The server failed to start up properly: javax.ejb.EJBException: java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] but the relevant property in rhq-server.properties IS set correctly: rhq.cassandra.seeds=fbry-test2.bc.jonqe.lab.eng.bos.redhat.com|7299|9142 I can see the error message from bz 950422 during the installation. When i kill the installation process and restart the rhq server: ./rhq-server.sh stop ./rhq-server.sh start the installation finishes successfully. So here is a possible cause: 1- installation process updates rhq-server.properties (rhq.cassandra.seeds) 2- exception from bz 950422 prevents the rhq server from reloading its updated configuration (which already contains correct rhq.cassandra.seeds) 3- rhq server uses the old configuration with empty rhq.cassandra.seeds property 4- exception java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] is thrown during the installation to the rhq server log. Version-Release number of selected component (if applicable): Version: 4.8.0-SNAPSHOT Build Number: 9c0489c How reproducible: 3 out of 3 Steps to Reproduce: 1. have a clean environment 2. create /var/lib/rhq with relevant permissions for non-root user 3. unzip rhq-server-4.8.0-SNAPSHOT.zip 4. cd rhq-server-4.8.0-SNAPSHOT/bin/ 5. chmod +x rhqctl 6. ./rhq-server.sh start (this step is a workaround for bz 967525) 7. ./rhqctl install Actual results: installation process (i can see it via 'ps -ef | grep java') hangs on message: "06:45:10,662 INFO [org.rhq.enterprise.server.installer.Installer] Installation is complete. The server should be ready shortly. and server installation fails with exception in rhq server log: The server failed to start up properly: javax.ejb.EJBException: java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] Expected results: Installation is successful Additional info: complete rhq server log attached installer exception: 06:43:47,884 WARN [org.rhq.enterprise.server.installer.InstallerServiceImpl] reloadConfiguration failed - restart the server to complete the installation: java.io.IOException: java.util.concurrent.ExecutionException: Operation failed at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:81) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:263) [rhq-jboss-as-dmr-client-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at org.rhq.common.jbossas.client.controller.CoreJBossASClient.reload(CoreJBossASClient.java:373) [rhq-jboss-as-dmr-client-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at org.rhq.common.jbossas.client.controller.CoreJBossASClient.reload(CoreJBossASClient.java:354) [rhq-jboss-as-dmr-client-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at org.rhq.enterprise.server.installer.InstallerServiceImpl.reloadConfiguration(InstallerServiceImpl.java:1145) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:303) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:115) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at org.rhq.enterprise.server.installer.Installer.main(Installer.java:56) [rhq-installer-util-4.8.0-SNAPSHOT.jar:4.8.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_24] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.6.0_24] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_24] at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_24] at org.jboss.modules.Module.run(Module.java:262) [jboss-modules.jar:1.2.0.CR1] at org.jboss.modules.Main.main(Main.java:329) [jboss-modules.jar:1.2.0.CR1] Caused by: java.util.concurrent.ExecutionException: Operation failed at org.jboss.threads.AsyncFutureTask.operationFailed(AsyncFutureTask.java:74) at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:268) at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:100) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] ... 14 more Caused by: java.io.IOException: Channel closed at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleChannelClosed(AbstractMessageHandler.java:381) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.as.controller.client.impl.RemotingModelControllerClient$2.handleClose(RemotingModelControllerClient.java:133) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.as.controller.client.impl.RemotingModelControllerClient$2.handleClose(RemotingModelControllerClient.java:130) [jboss-as-controller-client-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54) at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501) at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406) at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277) at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:515) at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:153) at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleEnd(ManagementChannelReceiver.java:129) [jboss-as-protocol-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4] at org.jboss.remoting3.remote.RemoteConnectionChannel$2.run(RemoteConnectionChannel.java:277) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24] at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]
The exception here is coming from the server installer. There was a regression in which we were using the wrong credentials to login into Cassandra and create the schema. The fix for this is available as of build 2292 of the rhq-master Jenkins job. Moving this to ON_QA since the fix is available.
Update: Version: 4.8.0-SNAPSHOT Build Number: bf44b14 - Installer exception (Caused by: java.io.IOException: Channel closed) is fixed. - Installer process doesn't hang any more. - Exception from rhq server log The server failed to start up properly: javax.ejb.EJBException: java.lang.IllegalArgumentException: Expected string of the form, hostname|jmxPort|nativeTransportPort: [] is still there and server is still not accessible from browser. Adding a workaround again (it is a bit hidden in the original description): ./rhq-server.sh stop ./rhq-server.sh start
Lowering severity. This is no longer blocking issue for our automation.