Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1435706

Summary: Unable to Upgrade Metrics due to missging compaction strategy class's
Product: OpenShift Container Platform Reporter: Eric Rich <erich>
Component: HawkularAssignee: John Sanda <jsanda>
Status: CLOSED ERRATA QA Contact: Peng Li <penli>
Severity: high Docs Contact:
Priority: high    
Version: 3.4.0CC: aos-bugs, jsanda, mwringe, nnosenzo, tkimura
Target Milestone: ---   
Target Release: 3.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: OCP 3.3 uses Cassandra 2.2.7 which does not ship with TimeWindowCompactionStrategy (TWCS). We back ported TWCS to OCP 3.3 to address some compaction related problems. OCP 3.4 uses Cassandra 3.0.9 which does ship with TWCS; however, class names have changed. When Cassandra starts, it cannot find the TWCS classes that were being used in OCP 3.3. Consequence: Cassandra fails to start after upgrading from OCP 3.3 to 3.4 Fix: The older version of TWCS is now packaged with Cassandra in OCP 3.4. Result: Cassandra starts up, and Hawkular Metrics is able to apply schema updates as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-04 14:29:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eric Rich 2017-03-24 14:58:07 UTC
Description of problem:

After upgrading the OCP metrics, cassandra fails to start with the following: 

INFO  12:30:29 Detected version upgrade from 2.2.7 to 3.0.9, snapshotting system keyspace
WARN  12:30:52 The sstable_compression option has been deprecated. You should use class instead
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Unable to find compaction strategy class 'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy'
org.apache.cassandra.exceptions.ConfigurationException: Unable to find compaction strategy class 'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy'


Version-Release number of selected component (if applicable): 3.4
How reproducible: Unknown

Steps to Reproduce:
1. Upgrade metrics from 3.3 to 3.4 

Additional info:

This may be related to changes that happened with https://bugzilla.redhat.com/show_bug.cgi?id=1416850 or https://issues.apache.org/jira/browse/CASSANDRA-9666 (version changes)

Comment 12 Peng Li 2017-04-01 06:54:21 UTC
Verified with latest 3.3.1 and 3.4.1 Metrics images. Attach 3.3.1 and 3.4.1 Cassandra logs

Steps:

1. deploy 3.3.1 Metrics and verify pods running well

2. refresh Metrics stack to 3.4.1

#curl -o metrics-deployer.yaml https://raw.githubusercontent.com/openshift/openshift-ansible/release-1.4/roles/openshift_hosted_templates/files/v1.4/enterprise/metrics-deployer.yaml

#oadm policy add-role-to-user view system:serviceaccount:openshift-infra:hawkular -n openshift-infra

#oc new-app -f metrics-deployer.yaml --as=system:serviceaccount:openshift-infra:metrics-deployer \
-p IMAGE_PREFIX=$PREFIX \
-p IMAGE_VERSION=3.4.1 \
-p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.$SUBDOMAIN \
-p MODE=refresh \
-p USE_PERSISTENT_STORAGE=true \
-p DYNAMICALLY_PROVISION_STORAGE=false \
-p CASSANDRA_NODES=1 \
-p CASSANDRA_PV_SIZE=10Gi \
-p USER_WRITE_ACCESS=false

3. check pods/pvc status

# oc get pvc
NAME                  STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
metrics-cassandra-1   Bound     pv2       10Gi       RWO           30m

# oc get pod
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-dogsq   1/1       Running     0          15m
hawkular-metrics-y4lov       1/1       Running     0          7m
heapster-42cfc               1/1       Running     0          7m
metrics-deployer-5ufwp       0/1       Completed   0          15m

4. check web console, the data before Metrics refresh can still show on UI.

***********************************************************************************************
Version:
#3.3.1
metrics-deployer           3.3.1               8c21385fb474        13 hours ago        760.2 MB
metrics-hawkular-metrics   3.3.1               06e96b34b801        13 hours ago        1.773 GB
metrics-cassandra          3.3.1               9baf309f1f4c        13 hours ago        531.5 MB
metrics-heapster           3.3.1               45c46225d986        13 hours ago        277.8 MB

#3.4.1
metrics-cassandra          3.4.1               938fabd788a7        9 hours ago         539.3 MB
metrics-deployer           3.4.1               3b1261351434        13 hours ago        896.3 MB
metrics-hawkular-metrics   3.4.1               486fcc7f0aa2        13 hours ago        1.26 GB
metrics-heapster           3.4.1               81f275debf15        13 hours ago        317.8 MB

Comment 16 errata-xmlrpc 2017-04-04 14:29:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0865