Bug 1157480

Summary: Upgrade of standalone storage node fails with relation "rhq_storage_node" does not exist
Product: [JBoss] JBoss Operations Network Reporter: Filip Brychta <fbrychta>
Component: Storage Node, UpgradeAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Filip Brychta <fbrychta>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: JON 3.3.0CC: mfoley
Target Milestone: CR01   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 14:03:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
upgrade output none

Description Filip Brychta 2014-10-27 10:22:00 UTC 
Created attachment 950931 [details]
upgrade output

Description of problem:
I have following setup (all using JON3.2.0.GA):
- server1: JON server (master - postgres db is here) + SN + agent
- server2: JON server (slave) + SN + agent
- server3: JON server (slave) + agent
- server4: SN + agent

following upgrade manual https://docs.jboss.org/author/display/RHQ/Upgrading+RHQ
upgrade of standalone SN failed with relation "rhq_storage_node" does not exist

Version-Release number of selected component (if applicable):
JON330ER05

How reproducible:
1/1

Steps to Reproduce:
1. prepare setup described above (having just server1 and server4 should be enough) 
2. stop all components (./rhqctl stop) on all servers
3. upgrade server4 (jon-server-3.3.0.ER05/bin/rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA/)

Actual results:
Exception in thread "main" java.lang.RuntimeException: Unable to update Storage Node [fbr-ha-st.bc.jonqe.lab.eng.bos.redhat.com] to version [4.12.0.JON330ER05]
	at org.rhq.storage.installer.StorageInstaller.updateStorageNodeVersion(StorageInstaller.java:1085)
	at org.rhq.storage.installer.StorageInstaller.stampStorageNodeVersion(StorageInstaller.java:1055)
	at org.rhq.storage.installer.StorageInstaller.run(StorageInstaller.java:283)
	at org.rhq.storage.installer.StorageInstaller.main(StorageInstaller.java:1130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.jboss.modules.Module.run(Module.java:312)
	at org.jboss.modules.Main.main(Main.java:460)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "rhq_storage_node" does not exist
  Position: 15
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
	at org.rhq.core.db.DatabaseType.checkColumnExists(DatabaseType.java:249)
	at org.rhq.core.db.upgrade.StorageNodeVersionColumnUpgrader.upgrade(StorageNodeVersionColumnUpgrader.java:47)
	at org.rhq.storage.installer.StorageInstaller.updateStorageNodeVersion(StorageInstaller.java:1078)
	... 9 more

Expected results:
According to https://bugzilla.redhat.com/show_bug.cgi?id=1139735#c3 (for this first release we may require that a Server be upgraded prior to a
  standalone SN, to get the new version-stamp fields into the RDB) it's probably necessary to upgrade JON server first.
So if it is really necessary, the upgrade process should fail gracefully and user should be informed that he needs to upgrade the JON server first.

Additional info:
complete upgrade log attached
Comment 1 Jay Shaughnessy 2014-10-27 14:09:09 UTC 
This looks like the result of a false assumption.  The latest code assumes the legacy rhq-server.properties file contains valid DB properties and this will be true for Storage Nodes co-located with a Server. But, for standalone SNs it was never required, or needed, to properly set the RDB properties.  This is no longer true.

We need to do two things:

1) Update the install and upgrade documentation.
2) Have the upgrade code test the DB connection prior to running.
Comment 2 Filip Brychta 2014-10-27 20:33:51 UTC 
I tried to set valid DB properties in rhq-server.properties before the upgrade but it didn't help.

I have following setup:
server1: JON server (master - postgres db is here) + SN + agent
server2: SN + agent


1- stop everything except storage nodes
2- server2: set correct DB properties in jon-server-3.3.0.ER05/bin/rhq-server.properties
3- jon-server-3.3.0.ER05/bin/rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA

Result:
The same error:
Caused by: org.postgresql.util.PSQLException: ERROR: relation "rhq_storage_node" does not exist
Comment 3 Jay Shaughnessy 2014-10-28 18:04:13 UTC 
Did you update the rhq-server.properties file in the --from-server-dir or the newly unzipped dir?  With the current code, if you updated the properties file in the --from-server-dir then it would not work, as above.  If you fixed it in the newly unzipped location it should work.

In the future code , after the fixes are in, you *will* need to update the --from-server-dir version (the old version of the properties file). That will be a one-time fix for this upgrade, and going forward it will get carried forward as expected.
Comment 4 Filip Brychta 2014-10-29 08:18:34 UTC 
I updated the rhq-server.properties file in the newly unzipped dir (comment 2, step 2 - set correct DB properties in jon-server-3.3.0.ER05/bin/rhq-server.properties)

I will double check that remote DB connection works
Comment 5 Filip Brychta 2014-10-29 09:46:38 UTC 
Remote DB connections work, it's possible to connect from server2 using psql.
Comment 6 Jay Shaughnessy 2014-10-29 14:15:01 UTC 
commit 1b45c2af77fe3a4670bc4317508780f80ae3f0bc
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Oct 29 09:54:13 2014 -0400

    [1139735, 1157480] More upgrade use cases that didn't work.
    [1139735] An upgrade with a number of storage nodes > than our replication
    factor (>= 4) could fail because our check for the storage cluster
    version can fail if the auth info or version info are not replicated
    to the currently running SNs.  To avoid this, for ugrades we now push
    all storage cluster interaction to the post-upgrade step (i.e.
    rhqctl --storage-schema).  This includes the schema creation for a new,
    remote storage cluster.

    [1157480] An upgrade of standalone storage node can fail because those
    installs may not have set proper DB props in rhq-server.properties.  It
    wasn't required in the past but it is now.  In the code we now make sure
    the properties are copied forward on upgrade. But there is also doco neede
    here because prior to upgrade from an earlier version, using standalone
    SNs, the old rhq-server.properties files will need to be updated.

    Also, update the --list-versions report to better reflect that it may need
    all SNs to be running to perform the storage schema version check.



release/jon3.3.x commit 8999436f3bfdc8bb7c311d1f27615c69d455f845
Author: Jay Shaughnessy <jshaughn>
Date:   Wed Oct 29 09:54:13 2014 -0400

    (cherry picked from commit 1b45c2af77fe3a4670bc4317508780f80ae3f0bc)
    Signed-off-by: Jay Shaughnessy <jshaughn>
Comment 7 Jay Shaughnessy 2014-10-30 16:36:55 UTC 
master commit 8d200b8d2c03a1b8566680d6e92fb64dde26fd7c
Author: Jay Shaughnessy <jshaughn>
Date:   Thu Oct 30 12:33:27 2014 -0400

    [1157480] resolve a test issue by better supporting a missing
    rhq-server.properties file if the --no-version-stamp option is specified.



release/jon3.3.x commit a610ad5f2ec0e4ad8c667d95c3db47a9279455b3
Author: Jay Shaughnessy <jshaughn>
Date:   Thu Oct 30 12:33:27 2014 -0400

    (cherry picked from commit 8d200b8d2c03a1b8566680d6e92fb64dde26fd7c)
    Signed-off-by: Jay Shaughnessy <jshaughn>
Comment 8 Simeon Pinder 2014-11-03 19:03:31 UTC 
Moving to ON_QA as available to test with latest brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=396547
Comment 9 Filip Brychta 2014-11-04 11:08:20 UTC 
Verified on
Version :	
3.3.0.CR01
Build Number :	
08c2f39:6ac97ac