Bug 1001128

Summary: Unable to install rhq server with rhq.autoinstall.database=overwrite
Product: [Other] RHQ Project Reporter: Filip Brychta <fbrychta>
Component: InstallerAssignee: Stefan Negrea <snegrea>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: high    
Version: 4.8CC: fbrychta, hrupp
Target Milestone: ER01   
Target Release: RHQ 4.9   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-24 19:08:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Filip Brychta 2013-08-26 14:28:34 UTC
Description of problem:
When the rhq.autoinstall.database property in rhq-server.properties is set to 'overwrite', installation fails with following exception:
Could not complete Cassandra schema installation: Authentication error on host /10.16.23.61: Username and/or password are incorrect

Version-Release number of selected component (if applicable):
Version: 4.9.0-SNAPSHOT
Build Number: d428376

How reproducible:
Always

Steps to Reproduce:
1. unzip rhq-server-4.9.0-SNAPSHOT.zip
2. set rhq.autoinstall.database property in rhq-server.properties to overwrite (rhq.autoinstall.database=overwrite)
3. ./rhqctl install

Actual results:
09:54:26,185 INFO  [org.rhq.enterprise.server.installer.InstallerServiceImpl] Cassandra schema exists but installer was told to overwrite it - a the existing  schema will be created now.
09:54:26,198 INFO  [org.rhq.cassandra.schema.VersionManager] Preparing to drop storage schema.
09:54:26,199 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections
09:54:26,200 INFO  [org.rhq.cassandra.schema.AbstractManager] Initializing storage node session.
09:54:26,427 INFO  [org.rhq.cassandra.schema.AbstractManager] Shutting down existing cluster connections
09:54:26,428 ERROR [org.rhq.enterprise.server.installer.InstallerServiceImpl] Could not complete Cassandra schema installation: Authentication error on host /10.16.23.61: Username and/or password are incorrect: java.lang.RuntimeException: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.16.23.61: Username and/or password are incorrect
	at org.rhq.cassandra.schema.VersionManager.drop(VersionManager.java:218) [rhq-cassandra-schema-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.cassandra.schema.SchemaManager.drop(SchemaManager.java:119) [rhq-cassandra-schema-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.prepareDatabase(InstallerServiceImpl.java:488) [rhq-installer-util-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.InstallerServiceImpl.install(InstallerServiceImpl.java:260) [rhq-installer-util-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.Installer.doInstall(Installer.java:115) [rhq-installer-util-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.enterprise.server.installer.Installer.main(Installer.java:56) [rhq-installer-util-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_24]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.6.0_24]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_24]
	at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_24]
	at org.jboss.modules.Module.run(Module.java:262) [jboss-modules.jar:1.2.0.CR1]
	at org.jboss.modules.Main.main(Main.java:329) [jboss-modules.jar:1.2.0.CR1]
Caused by: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host /10.16.23.61: Username and/or password are incorrect
	at com.datastax.driver.core.Connection.initializeTransport(Connection.java:168) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Connection.<init>(Connection.java:129) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Connection.<init>(Connection.java:56) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Connection$Factory.open(Connection.java:412) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:205) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:168) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:81) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:644) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:586) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Cluster.<init>(Cluster.java:69) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Cluster.buildFrom(Cluster.java:96) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at com.datastax.driver.core.Cluster$Builder.build(Cluster.java:567) [cassandra-driver-core-1.0.2-rhq-1.2.4.jar:]
	at org.rhq.cassandra.util.ClusterBuilder.build(ClusterBuilder.java:111) [rhq-cassandra-util-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.cassandra.schema.AbstractManager.initClusterSession(AbstractManager.java:114) [rhq-cassandra-schema-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.cassandra.schema.AbstractManager.initClusterSession(AbstractManager.java:100) [rhq-cassandra-schema-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	at org.rhq.cassandra.schema.VersionManager.drop(VersionManager.java:213) [rhq-cassandra-schema-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
	... 11 more

Expected results:
No errors

Additional info:
Cassandra credentials are still set to default cassandra. I'm able to connect using './cqlsh fbry-test.bc.jonqe.lab.eng.bos.redhat.com -u cassandra -p cassandra' but not able to connect using './cqlsh fbry-test.bc.jonqe.lab.eng.bos.redhat.com -u rhqadmin -p rhqadmin'

Comment 1 Heiko W. Rupp 2013-08-28 20:00:34 UTC
Filip, is that a brand new install without an existing C* instance?
Or an "upgrade" that is trying to re-use the existing C* install by overwriting it?

Comment 2 John Sanda 2013-08-28 21:13:29 UTC
Stefan, I am assigning this over to you. I looked at the drop XML scripts in the schema manager. There is no reason to revert the password of the cassandra user. In fact if anything, we really ought to drop the default cassandra user. The reason we do not delete it goes back to a misunderstanding on my part about bootstrapping C* nodes. At the time I was doing it incorrectly, and while debugging I thought that maybe deleting the cassandra user was causing problems. I also do not think we should be dropping the rhqadmin/internal user. All we should be doing there is dropping the rhq schema. 

Not sure what all is involved with this issue but I figured you would be better suited to look at it.

Comment 3 Filip Brychta 2013-08-29 07:40:24 UTC
(In reply to Heiko W. Rupp from comment #1)
> Filip, is that a brand new install without an existing C* instance?
> Or an "upgrade" that is trying to re-use the existing C* install by
> overwriting it?

Brand new install without existing C* instance.

Comment 4 Stefan Negrea 2013-08-30 17:29:31 UTC
The issue here was that the drop procedure would fail when the schema was not installed at all. The drop procedure attempts to first re-establish the generic Cassandra user/pass before getting the cluster in a pre-RHQ state. The drop code was wrongly throwing an error if the RHQ user and schema did not exist. 

The fix catches the first authentication error and takes no action. The drop code will then attempt to establish a Cassandra connection with the generic user name and password and continue the drop procedure. A second authentication is considered an error and will visible to the user.

master commit:
https://git.fedorahosted.org/cgit/rhq/rhq.git/commit/?id=6eb9a98f7316382cd070922402fc1d9957e41076

Comment 5 Filip Brychta 2013-09-02 08:21:38 UTC
Verified on
Version: 4.9.0-SNAPSHOT
Build Number: a9245fd

Comment 6 Heiko W. Rupp 2013-09-24 19:08:38 UTC
Bulk closing of RHQ 4.9 verified items