Bug 1417729 - Cassandra error starting up due to "mutation checksum failure" on a commit log
Summary: Cassandra error starting up due to "mutation checksum failure" on a commit log
Keywords:
Status: CLOSED DUPLICATE of bug 1385427
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.2.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Matt Wringe
QA Contact: Peng Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-30 19:07 UTC by Eric Jones
Modified: 2020-03-11 15:41 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-30 23:03:05 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Eric Jones 2017-01-30 19:07:32 UTC
Description of problem:
Customer had been running metrics for ~5-6 months and then suddenly metrics were no longer available.

After looking into the logs they saw the following message:

ERROR <TIME> Exiting due to error while processing commit log during
initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Mutation checksum failure at 27357160 in CommitLog-5-1484574169950.log

After deleting this commitlog (they have saved it and I will provide it in another update) cassandra was able to start up normally and metrics started back up properly.

Version-Release number of selected component (if applicable):
metrics-cassandra:3.2.1
openshift v3.2.1.13-1-gc2a90e1
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

Comment 2 Matt Wringe 2017-01-30 23:03:05 UTC
This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1385427

This issue is already resolved in 3.2.1, but it requires that the templates are updated, which would occur during a new metrics install.

You can either deploy metrics again which will update the Cassandra template to fix this, or manually run the following command to give Cassandra more time to process its commit files when its being shut down:

$ oc patch rc hawkular-cassandra-1 -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":"1800"}}}}'

If you wish to skip over the commit log failures in the future, you can also run the following command:

oc patch rc hawkular-cassandra-1 -p '{"spec":{"template":{"spec":{"containers":[{"name":"hawkular-cassandra-1", "env": [{"name": "JVM_OPTS", "value":"-Dcassandra.commitlog.ignorereplayerrors=true"}]}]}}}}'

*** This bug has been marked as a duplicate of bug 1385427 ***


Note You need to log in before you can comment on or make changes to this bug.