Created attachment 734287 [details] storage_installer_log Description of problem: Cassandra Storage installer - ./rhq-storage-installer.sh doesn't show any Warning or Error when cassandra is not started. If another instance of cassandra is started with the same jmx port, or another process is started with the same jmx port as storage installer would use to start cassandra, there is no warning or error at the end of script run telling that cassandra was installed but not started. Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. download latest rhq server installer (having cassandra integrated) 2. run rhq-storage-installer.sh when default 7299 jmx port is busy 3. Actual results: storage node is installed and not started - no error or warning is shown Expected results: storage node is installed and not started - there is an error or warning telling that cassandra server was not started Additional info: Log messages are attached
I started an instance of Cassandra that was listening on port 7200 and then ran the installer specifying 7200 for the JMX port. I did get a warning message from the installer. Here is the command line along with the output. localhost:bin jsanda$ ./rhq-storage-installer.sh --commitlog=../storage/commit_log --data=../storage/data --saved-caches=../storage/saved_caches -j 7200 Starting RHQ Storage Installer ... 12:12:12,299 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA 12:12:12,361 INFO [org.rhq.storage.installer.StorageInstaller] Running RHQ Storage Node installer... 12:12:12,372 INFO [org.rhq.cassandra.Deployer] Unzipping storage node to /Users/jsanda/Development/redhat/rhq/modules/enterprise/server/appserver/target/rhq-server-4.7.0-SNAPSHOT/storage 12:12:12,600 INFO [org.rhq.cassandra.Deployer] Applying configuration changes to /Users/jsanda/Development/redhat/rhq/modules/enterprise/server/appserver/target/rhq-server-4.7.0-SNAPSHOT/storage/conf/cassandra.yaml 12:12:12,654 INFO [org.rhq.cassandra.Deployer] Applying configuration changes to /Users/jsanda/Development/redhat/rhq/modules/enterprise/server/appserver/target/rhq-server-4.7.0-SNAPSHOT/storage/conf/log4j-server.properties 12:12:12,657 INFO [org.rhq.cassandra.Deployer] Applying configuration changes to /Users/jsanda/Development/redhat/rhq/modules/enterprise/server/appserver/target/rhq-server-4.7.0-SNAPSHOT/storage/conf/cassandra-env.sh 12:12:12,670 INFO [org.rhq.cassandra.Deployer] Updating file permissions in /Users/jsanda/Development/redhat/rhq/modules/enterprise/server/appserver/target/rhq-server-4.7.0-SNAPSHOT/storage/bin 12:12:12,672 INFO [org.rhq.storage.installer.StorageInstaller] Finished installing RHQ Storage Node. 12:12:12,672 INFO [org.rhq.storage.installer.StorageInstaller] Updating rhq-server.properties... 12:12:12,677 INFO [org.rhq.storage.installer.StorageInstaller] Starting RHQ Storage Node 12:12:12,744 INFO [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused 12:12:12,745 INFO [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 3000 ms... 12:12:15,747 INFO [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused 12:12:15,748 INFO [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 6000 ms... 12:12:21,751 INFO [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused 12:12:21,752 INFO [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 9000 ms... 12:12:30,753 INFO [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused 12:12:30,754 INFO [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 12000 ms... 12:12:42,756 INFO [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused 12:12:42,757 INFO [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 15000 ms... 12:12:57,758 WARN [org.rhq.storage.installer.StorageInstaller] Could not verify that the node is up and running. 12:12:57,759 INFO [org.rhq.storage.installer.StorageInstaller] Installation of the storage node is complete. Note the WARN message at the end which says it could not verify that the node is up. There is room for improvement here though. The installer can capture the exit code from the Cassandra script, and if it is non-zero, then the installer can display any startup errors reported by Cassandra. For example, if I try to start the instance that I just installed, here's what happens, localhost:bin jsanda$ ./cassandra -f xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4096M -Xmx4096M -Xmn800M -XX:+HeapDumpOnOutOfMemoryError Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7200; nested exception is: java.net.BindException: Address already in use The installer should log that error message. I will make those changes.
I have introduced a couple changes to the installer. When starting Cassandra it now checks for and logs anything Cassandra writes to stderr. If there are error messages, the isntaller will not perform the check to verify that Cassandra is up in running. Instead the a detailed message, in addition to the error message(s) from Cassandra, is logged. And since we know what will happen when there is a conflict with the JMX port, the installer inspects the error message and will displayed an even more detailed message if the errors mentions a port conflict. Here is output from the installer when there is a conflict with the JMX port: 1:29:58,647 INFO [org.rhq.storage.installer.StorageInstaller] Updating rhq-server.properties... 21:29:58,662 INFO [org.rhq.storage.installer.StorageInstaller] Starting RHQ Storage Node 21:29:59,333 ERROR [org.rhq.storage.installer.StorageInstaller] The storage node reported the following errors while trying to start: Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7200; nested exception is: java.net.BindException: Address already in use 21:29:59,334 ERROR [org.rhq.storage.installer.StorageInstaller] This error may indicate a conflict for the JMX port. 21:29:59,335 ERROR [org.rhq.storage.installer.StorageInstaller] Please review your configuration for possible sources of errors such as port conflicts or invalid arguments/options passed to the java executable. 21:29:59,335 ERROR [org.rhq.storage.installer.StorageInstaller] The installer will now exit. feature/cassandra-backend commit hash: dd391e5095b25
verified. Thank you
Created attachment 735932 [details] storage_installer-log_1.txt verified! verification log.