Bug 983226

Summary: An error occurred while running the storage node upgrade: Process exited with an error: 1 during rhqctl upgrade from 4.8 to 4.9
Product: [JBoss] JBoss Operations Network Reporter: Armine Hovsepyan <ahovsepy>
Component: UpgradeAssignee: John Sanda <jsanda>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: JON 3.2CC: jsanda, mfoley
Target Milestone: ER01   
Target Release: JON 3.2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-02 20:33:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 951619    
Attachments:
Description Flags
rhqctl.log
none
rhqctl-upgrade-4.8-4.9.log none

Description Armine Hovsepyan 2013-07-10 18:47:37 UTC
Created attachment 771771 [details]
rhqctl.log

Description of problem:
An error occurred while running the storage node upgrade: Process exited with an error: 1  during rhqctl upgrade from 4.8 to 4.9

Version-Release number of selected component (if applicable):
http://sourceforge.net/projects/rhq/files/rhq/rhq-4.8.0/
4.9 master build 2414

How reproducible:
2 out of 2

Steps to Reproduce:
1. run rhqctl install for 4.8
2. run ./rhq-server-4.9.0-SNAPSHOT/bin/rhqctl upgrade --from-server-dir=/home/hudson/rhq-server-4.8.0



Actual results:
An error occurred while running the storage node upgrade: Process exited with an error: 1  

Expected results:
rhq server with storage and agent is upgraded


Additional info:
log attached

Comment 1 John Sanda 2013-07-17 19:21:24 UTC
I introduced this regression with a change I made in the storage installer to look for the new cassandra-jvm.properties file instead of cassandra-env.sh. The JMX port is defined in cassandra-env.sh in RHQ 4.8. The port is defined in cassandra-jvm.properties in RHQ 4.9 and going forward. The issue is that cassandra-jvm.properties does not exist in 4.8.0. I need to update the installer upgrade code to migrate from cassandra-env.sh to cassandra-jvm.properties.

Comment 2 John Sanda 2013-07-19 17:52:21 UTC
The storage installer has been updated along with some automated tests so that correctly handles both cassandra-env.sh (for 4.8 installs) and cassandra-jvm.properties for later installs. The fix will be in build 2443 of the rhq-master Jenkins job.

master branch commit hash: 656cc606625

Comment 3 Armine Hovsepyan 2013-07-22 10:08:27 UTC
Upgrade is not working. Getting the following exception:
 "java.lang.RuntimeException: org.apache.cassandra.exceptions.ConfigurationException: Cannot add column family 'users' to non existing keyspace 'system_auth'"

Please get log attached.

Comment 4 Armine Hovsepyan 2013-07-22 10:09:13 UTC
Created attachment 776845 [details]
rhqctl-upgrade-4.8-4.9.log

Comment 5 John Sanda 2013-07-22 18:27:02 UTC
I am not exactly sure of the cause for that error but I suspect it is related to the fact that you cannot directly upgrade from a RHQ 4.8 install. This is because of the removal of native libraries from Cassandra that were present in 4.8. Because those libraries were packaged with Cassandra in RHQ 4.8, table compression may have been used if the native library support was available. Upgrading from 4.8 requires running a patch script against your 4.8 installation. I need to write up some docs on that script. I will update this bug with the link to those docs when I have them ready.

Comment 6 John Sanda 2013-07-22 19:22:22 UTC
Some instructions will be included in the RHQ 4.9 release notes which have not yet been created. For now, you run the patch script directly from the source tree in the master branch. The script is,

<rhq-source>/etc/scripts/rhq48-storage-patch/rhq48-storage-patch.sh

Instructions are provided as comments at the top of the script.

Comment 7 Armine Hovsepyan 2013-07-23 11:09:17 UTC
Verified.

storage patch worked, upgrade was possible from 4.8 to 4.9 after that. 
Could you please include the patch in rhq archive?

Thanks in advance.