Bug 1435706 - Unable to Upgrade Metrics due to missging compaction strategy class's
Summary: Unable to Upgrade Metrics due to missging compaction strategy class's
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.4.z
Assignee: John Sanda
QA Contact: Peng Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-24 14:58 UTC by Eric Rich
Modified: 2020-08-13 08:59 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: OCP 3.3 uses Cassandra 2.2.7 which does not ship with TimeWindowCompactionStrategy (TWCS). We back ported TWCS to OCP 3.3 to address some compaction related problems. OCP 3.4 uses Cassandra 3.0.9 which does ship with TWCS; however, class names have changed. When Cassandra starts, it cannot find the TWCS classes that were being used in OCP 3.3. Consequence: Cassandra fails to start after upgrading from OCP 3.3 to 3.4 Fix: The older version of TWCS is now packaged with Cassandra in OCP 3.4. Result: Cassandra starts up, and Hawkular Metrics is able to apply schema updates as expected.
Clone Of:
Environment:
Last Closed: 2017-04-04 14:29:14 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0865 0 normal SHIPPED_LIVE OpenShift Container Platform 3.4.1.12, 3.3.1.17-4, and 3.2.1.30 bug fix update 2017-04-04 18:27:43 UTC

Description Eric Rich 2017-03-24 14:58:07 UTC 
Description of problem:

After upgrading the OCP metrics, cassandra fails to start with the following: 

INFO  12:30:29 Detected version upgrade from 2.2.7 to 3.0.9, snapshotting system keyspace
WARN  12:30:52 The sstable_compression option has been deprecated. You should use class instead
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: Unable to find compaction strategy class 'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy'
org.apache.cassandra.exceptions.ConfigurationException: Unable to find compaction strategy class 'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy'


Version-Release number of selected component (if applicable): 3.4
How reproducible: Unknown

Steps to Reproduce:
1. Upgrade metrics from 3.3 to 3.4 

Additional info:

This may be related to changes that happened with https://bugzilla.redhat.com/show_bug.cgi?id=1416850 or https://issues.apache.org/jira/browse/CASSANDRA-9666 (version changes)
Comment 12 Peng Li 2017-04-01 06:54:21 UTC 
Verified with latest 3.3.1 and 3.4.1 Metrics images. Attach 3.3.1 and 3.4.1 Cassandra logs

Steps:

1. deploy 3.3.1 Metrics and verify pods running well

2. refresh Metrics stack to 3.4.1

#curl -o metrics-deployer.yaml https://raw.githubusercontent.com/openshift/openshift-ansible/release-1.4/roles/openshift_hosted_templates/files/v1.4/enterprise/metrics-deployer.yaml

#oadm policy add-role-to-user view system:serviceaccount:openshift-infra:hawkular -n openshift-infra

#oc new-app -f metrics-deployer.yaml --as=system:serviceaccount:openshift-infra:metrics-deployer \
-p IMAGE_PREFIX=$PREFIX \
-p IMAGE_VERSION=3.4.1 \
-p HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.$SUBDOMAIN \
-p MODE=refresh \
-p USE_PERSISTENT_STORAGE=true \
-p DYNAMICALLY_PROVISION_STORAGE=false \
-p CASSANDRA_NODES=1 \
-p CASSANDRA_PV_SIZE=10Gi \
-p USER_WRITE_ACCESS=false

3. check pods/pvc status

# oc get pvc
NAME                  STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
metrics-cassandra-1   Bound     pv2       10Gi       RWO           30m

# oc get pod
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-dogsq   1/1       Running     0          15m
hawkular-metrics-y4lov       1/1       Running     0          7m
heapster-42cfc               1/1       Running     0          7m
metrics-deployer-5ufwp       0/1       Completed   0          15m

4. check web console, the data before Metrics refresh can still show on UI.

***********************************************************************************************
Version:
#3.3.1
metrics-deployer           3.3.1               8c21385fb474        13 hours ago        760.2 MB
metrics-hawkular-metrics   3.3.1               06e96b34b801        13 hours ago        1.773 GB
metrics-cassandra          3.3.1               9baf309f1f4c        13 hours ago        531.5 MB
metrics-heapster           3.3.1               45c46225d986        13 hours ago        277.8 MB

#3.4.1
metrics-cassandra          3.4.1               938fabd788a7        9 hours ago         539.3 MB
metrics-deployer           3.4.1               3b1261351434        13 hours ago        896.3 MB
metrics-hawkular-metrics   3.4.1               486fcc7f0aa2        13 hours ago        1.26 GB
metrics-heapster           3.4.1               81f275debf15        13 hours ago        317.8 MB
Comment 16 errata-xmlrpc 2017-04-04 14:29:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0865
Note You need to log in before you can comment on or make changes to this bug.