Red Hat Bugzilla – Bug 1019823
Installing 4.9: Connection refused in server installer script
Last modified: 2013-11-05 04:58:33 EST
Description of problem:
When running the RHQ server installer with command:
rhqctl upgrade --from-server-dir=/opt/rhq/rhq-server.OLD
I get the following in the output and rhq-storage-installer.log and the RHQ server won't start.
14:08:55,695 INFO [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused
14:08:55,696 INFO [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 12000 ms...
This keeps coming until I abort the installation. Cassandra however is running and logging without exceptions.
Trying to start the RHQ server I get an error about the server not being installed.
[rhqadmin@d26apvl007 rhq-server]$ ./bin/rhqctl start --server
15:03:08,027 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1
15:03:08,265 WARN [org.rhq.server.control.command.Start] It appears that the server is not installed. The --server option will be ignored.
[rhqadmin@d26apvl007 rhq-server]$ ./bin/rhqctl status
15:03:45,388 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1
RHQ Storage Node (pid 10592 ) IS running
Version-Release number of selected component (if applicable):
RHQ Server 4.9 final
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Linux d26apvl007.test.local 2.6.32-358.14.1.el6.x86_64 #1 SMP Mon Jun 17 15:54:20 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
Steps to Reproduce:
1. Back up old installation to rhq-server.old
2. Do needed edits in rhq-server.properties
3. Run command rhqctl upgrade --from-server-dir=/opt/rhq/rhq-server.OLD
Cassandra is started but installer cannot contact it, and fails.
Expect that installer also installs the RHQ-server.
Upgrading from RHQ 4.9 SNAPSHOT, might be relevant
After it failed the first time, I decided I wanted to try starting with a "clean slate" so removed the RHQ storage data directory, mapped to /opt/rhq/rhq-storage.
This was after running the "rhq48-storage-patch".
Output from RHQ Storage patch run:
[rhqadmin@d26apvl007 rhq48-storage-patch]$ ./rhq48-storage-patch.sh /opt/rhq/rhq-server 127.0.0.1 9160 7299
Copying patch file to /opt/rhq/rhq-server/rhq-storage/lib
Starting RHQ Storage node
15:25:34,525 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1
RHQ storage node (pid 12246) is running
Waiting for RHQ Storage Node to start up...
Running CQL script to disable table compression
Traceback (most recent call last):
File "/opt/rhq/rhq-server/rhq-storage/bin/cqlsh", line 2279, in <module>
File "/opt/rhq/rhq-server/rhq-storage/bin/cqlsh", line 2265, in main
File "/opt/rhq/rhq-server/rhq-storage/bin/cqlsh", line 483, in __init__
File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/connection.py", line 143, in connect
File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/connection.py", line 59, in __init__
File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/thrifteries.py", line 157, in establish_connection
File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/cassandra/Cassandra.py", line 455, in login
File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/cassandra/Cassandra.py", line 476, in recv_login
cql.cassandra.ttypes.AuthenticationException: AuthenticationException(why='Username and/or password are incorrect')
Rebuilding data files for system keyspace
Rebuilding data files for system_traces keyspace
Rebuilding data files for system_auth keyspace
Rebuilding data files for rhq keyspace
Keyspace [rhq] does not exist.
Shutting down the RHQ Storage node
15:25:45,217 INFO [org.jboss.modules] JBoss Modules version 1.2.0.CR1
Trying to stop the RHQ Server...
RHQ Server (pid=12414) is stopping...
RHQ Server has stopped.
Stopping RHQ storage node...
RHQ storage node (pid=12246) is stopping...
kill 12246: No such process
RHQ storage node has stopped
Removing patch file
Table compression has been disabled for all keyspaces. You are now ready to upgrade your RHQ installation.
I am setting this to Closed as I was able to get 4.9 running using a clean Install instead of Update.
Not sure as to the cause and not able to replicate not that 4.9 is up.