Bug 1019823

Summary: Installing 4.9: Connection refused in server installer script
Product: [Other] RHQ Project Reporter: Stian Lund <stianlund+bugzilla>
Component: Core Server, InstallerAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED DEFERRED QA Contact: Mike Foley <mfoley>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.9CC: hrupp, stianlund+bugzilla
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-05 09:58:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Stian Lund 2013-10-16 13:10:23 UTC 
Description of problem:

When running the RHQ server installer with command:
rhqctl upgrade --from-server-dir=/opt/rhq/rhq-server.OLD
I get the following in the output and rhq-storage-installer.log and the RHQ server won't start.

14:08:55,695 INFO  [org.rhq.storage.installer.StorageInstaller] The storage node is not up: java.net.ConnectException: Connection refused
14:08:55,696 INFO  [org.rhq.storage.installer.StorageInstaller] Checking storage node status again in 12000 ms...

This keeps coming until I abort the installation. Cassandra however is running and logging without exceptions.

Trying to start the RHQ server I get an error about the server not being installed.

[rhqadmin@d26apvl007 rhq-server]$ ./bin/rhqctl start --server
15:03:08,027 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
15:03:08,265 WARN  [org.rhq.server.control.command.Start] It appears that the server is not installed. The --server option will be ignored.

[rhqadmin@d26apvl007 rhq-server]$ ./bin/rhqctl status
15:03:45,388 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
RHQ Storage Node               (pid 10592  ) IS running

Version-Release number of selected component (if applicable):
RHQ Server 4.9 final
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Linux d26apvl007.test.local 2.6.32-358.14.1.el6.x86_64 #1 SMP Mon Jun 17 15:54:20 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux


How reproducible:
Reproducible

Steps to Reproduce:
1. Back up old installation to rhq-server.old
2. Do needed edits in rhq-server.properties
3. Run command rhqctl upgrade --from-server-dir=/opt/rhq/rhq-server.OLD

Actual results:

Cassandra is started but installer cannot contact it, and fails.


Expected results:
Expect that installer also installs the RHQ-server.


Additional info:

Upgrading from RHQ 4.9 SNAPSHOT, might be relevant

After it failed the first time, I decided I wanted to try starting with a "clean slate" so removed the RHQ storage data directory, mapped to /opt/rhq/rhq-storage.

This was after running the "rhq48-storage-patch".
Comment 1 Stian Lund 2013-10-16 13:36:04 UTC 
Output from RHQ Storage patch run:

[rhqadmin@d26apvl007 rhq48-storage-patch]$ ./rhq48-storage-patch.sh /opt/rhq/rhq-server 127.0.0.1 9160 7299
Copying patch file to /opt/rhq/rhq-server/rhq-storage/lib
Starting RHQ Storage node
15:25:34,525 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
RHQ storage node (pid 12246) is running
Waiting for RHQ Storage Node to start up...
Running CQL script to disable table compression
Traceback (most recent call last):
  File "/opt/rhq/rhq-server/rhq-storage/bin/cqlsh", line 2279, in <module>
    main(*read_options(sys.argv[1:], os.environ))
  File "/opt/rhq/rhq-server/rhq-storage/bin/cqlsh", line 2265, in main
    display_float_precision=options.float_precision)
  File "/opt/rhq/rhq-server/rhq-storage/bin/cqlsh", line 483, in __init__
    cql_version=cqlver, transport=transport)
  File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/connection.py", line 143, in connect
  File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/connection.py", line 59, in __init__
  File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/thrifteries.py", line 157, in establish_connection
  File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/cassandra/Cassandra.py", line 455, in login
  File "/opt/rhq/rhq-server/rhq-storage/bin/../lib/cql-internal-only-1.4.0.zip/cql-1.4.0/cql/cassandra/Cassandra.py", line 476, in recv_login
cql.cassandra.ttypes.AuthenticationException: AuthenticationException(why='Username and/or password are incorrect')
Rebuilding data files for system keyspace
Rebuilding data files for system_traces keyspace
Rebuilding data files for system_auth keyspace
Rebuilding data files for rhq keyspace
Keyspace [rhq] does not exist.
Shutting down the RHQ Storage node
15:25:45,217 INFO  [org.jboss.modules] JBoss Modules version 1.2.0.CR1
Trying to stop the RHQ Server...
RHQ Server (pid=12414) is stopping...
RHQ Server has stopped.
Stopping RHQ storage node...
RHQ storage node (pid=12246) is stopping...
kill 12246: No such process
RHQ storage node has stopped
Removing patch file
Table compression has been disabled for all keyspaces. You are now ready to upgrade your RHQ installation.
Comment 2 Stian Lund 2013-11-05 09:58:33 UTC 
I am setting this to Closed as I was able to get 4.9 running using a clean Install instead of Update.

Not sure as to the cause and not able to replicate not that 4.9 is up.