Created attachment 950931 [details] upgrade output Description of problem: I have following setup (all using JON3.2.0.GA): - server1: JON server (master - postgres db is here) + SN + agent - server2: JON server (slave) + SN + agent - server3: JON server (slave) + agent - server4: SN + agent following upgrade manual https://docs.jboss.org/author/display/RHQ/Upgrading+RHQ upgrade of standalone SN failed with relation "rhq_storage_node" does not exist Version-Release number of selected component (if applicable): JON330ER05 How reproducible: 1/1 Steps to Reproduce: 1. prepare setup described above (having just server1 and server4 should be enough) 2. stop all components (./rhqctl stop) on all servers 3. upgrade server4 (jon-server-3.3.0.ER05/bin/rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA/) Actual results: Exception in thread "main" java.lang.RuntimeException: Unable to update Storage Node [fbr-ha-st.bc.jonqe.lab.eng.bos.redhat.com] to version [4.12.0.JON330ER05] at org.rhq.storage.installer.StorageInstaller.updateStorageNodeVersion(StorageInstaller.java:1085) at org.rhq.storage.installer.StorageInstaller.stampStorageNodeVersion(StorageInstaller.java:1055) at org.rhq.storage.installer.StorageInstaller.run(StorageInstaller.java:283) at org.rhq.storage.installer.StorageInstaller.main(StorageInstaller.java:1130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.modules.Module.run(Module.java:312) at org.jboss.modules.Main.main(Main.java:460) Caused by: org.postgresql.util.PSQLException: ERROR: relation "rhq_storage_node" does not exist Position: 15 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302) at org.rhq.core.db.DatabaseType.checkColumnExists(DatabaseType.java:249) at org.rhq.core.db.upgrade.StorageNodeVersionColumnUpgrader.upgrade(StorageNodeVersionColumnUpgrader.java:47) at org.rhq.storage.installer.StorageInstaller.updateStorageNodeVersion(StorageInstaller.java:1078) ... 9 more Expected results: According to https://bugzilla.redhat.com/show_bug.cgi?id=1139735#c3 (for this first release we may require that a Server be upgraded prior to a standalone SN, to get the new version-stamp fields into the RDB) it's probably necessary to upgrade JON server first. So if it is really necessary, the upgrade process should fail gracefully and user should be informed that he needs to upgrade the JON server first. Additional info: complete upgrade log attached
This looks like the result of a false assumption. The latest code assumes the legacy rhq-server.properties file contains valid DB properties and this will be true for Storage Nodes co-located with a Server. But, for standalone SNs it was never required, or needed, to properly set the RDB properties. This is no longer true. We need to do two things: 1) Update the install and upgrade documentation. 2) Have the upgrade code test the DB connection prior to running.
I tried to set valid DB properties in rhq-server.properties before the upgrade but it didn't help. I have following setup: server1: JON server (master - postgres db is here) + SN + agent server2: SN + agent 1- stop everything except storage nodes 2- server2: set correct DB properties in jon-server-3.3.0.ER05/bin/rhq-server.properties 3- jon-server-3.3.0.ER05/bin/rhqctl upgrade --from-server-dir /home/hudson/jon-server-3.2.0.GA Result: The same error: Caused by: org.postgresql.util.PSQLException: ERROR: relation "rhq_storage_node" does not exist
Did you update the rhq-server.properties file in the --from-server-dir or the newly unzipped dir? With the current code, if you updated the properties file in the --from-server-dir then it would not work, as above. If you fixed it in the newly unzipped location it should work. In the future code , after the fixes are in, you *will* need to update the --from-server-dir version (the old version of the properties file). That will be a one-time fix for this upgrade, and going forward it will get carried forward as expected.
I updated the rhq-server.properties file in the newly unzipped dir (comment 2, step 2 - set correct DB properties in jon-server-3.3.0.ER05/bin/rhq-server.properties) I will double check that remote DB connection works
Remote DB connections work, it's possible to connect from server2 using psql.
commit 1b45c2af77fe3a4670bc4317508780f80ae3f0bc Author: Jay Shaughnessy <jshaughn> Date: Wed Oct 29 09:54:13 2014 -0400 [1139735, 1157480] More upgrade use cases that didn't work. [1139735] An upgrade with a number of storage nodes > than our replication factor (>= 4) could fail because our check for the storage cluster version can fail if the auth info or version info are not replicated to the currently running SNs. To avoid this, for ugrades we now push all storage cluster interaction to the post-upgrade step (i.e. rhqctl --storage-schema). This includes the schema creation for a new, remote storage cluster. [1157480] An upgrade of standalone storage node can fail because those installs may not have set proper DB props in rhq-server.properties. It wasn't required in the past but it is now. In the code we now make sure the properties are copied forward on upgrade. But there is also doco neede here because prior to upgrade from an earlier version, using standalone SNs, the old rhq-server.properties files will need to be updated. Also, update the --list-versions report to better reflect that it may need all SNs to be running to perform the storage schema version check. release/jon3.3.x commit 8999436f3bfdc8bb7c311d1f27615c69d455f845 Author: Jay Shaughnessy <jshaughn> Date: Wed Oct 29 09:54:13 2014 -0400 (cherry picked from commit 1b45c2af77fe3a4670bc4317508780f80ae3f0bc) Signed-off-by: Jay Shaughnessy <jshaughn>
master commit 8d200b8d2c03a1b8566680d6e92fb64dde26fd7c Author: Jay Shaughnessy <jshaughn> Date: Thu Oct 30 12:33:27 2014 -0400 [1157480] resolve a test issue by better supporting a missing rhq-server.properties file if the --no-version-stamp option is specified. release/jon3.3.x commit a610ad5f2ec0e4ad8c667d95c3db47a9279455b3 Author: Jay Shaughnessy <jshaughn> Date: Thu Oct 30 12:33:27 2014 -0400 (cherry picked from commit 8d200b8d2c03a1b8566680d6e92fb64dde26fd7c) Signed-off-by: Jay Shaughnessy <jshaughn>
Moving to ON_QA as available to test with latest brew build: https://brewweb.devel.redhat.com//buildinfo?buildID=396547
Verified on Version : 3.3.0.CR01 Build Number : 08c2f39:6ac97ac