Bug 1596261
| Summary: | [free-stg] hawkular-cassandra pods in crashloopbackoff due to: Caused by: java.io.IOException: Compressed lengths mismatch - 3 bytes remain | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Justin Pierce <jupierce> | ||||
| Component: | Hawkular | Assignee: | John Sanda <jsanda> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Junqi Zhao <juzhao> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.10.0 | CC: | aos-bugs, jupierce | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 3.10.z | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-08-06 20:28:55 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
You need to delete the corrupt SSTable: $ rm /cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-* and then restart Cassandra. Did deleting the corrupt SSTable resovle the problem? Yes, the pods are running now. |
Created attachment 1455336 [details] Logs for crashing pod Description of problem: Noted that after upgrade to OCP v3.10.9 using openshift-ansible, hawkular-cassandra pods were in crash loop. Partial stack trace: Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-big-Data.db): corruption detected, chunk at 0 of length 47. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:187) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1] ... 24 common frames omitted Caused by: java.io.IOException: Compressed lengths mismatch - 3 bytes remain at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:124) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1] Version-Release number of selected component (if applicable): v3.10.9 Additional info: See attachments for listings.