Bug 1462461

Summary: Cassandra in CrashLoopBackOff with sstables error message
Product: OpenShift Container Platform Reporter: Steven Walter <stwalter>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED NOTABUG QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5.0CC: aos-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-17 22:32:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steven Walter 2017-06-17 20:52:39 UTC
Description of problem:
Cassandra pod in CrashLoopBackOff. Cassandra logs has message like:

ERROR 20:28:24 Detected unreadable sstables /cassandra_data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-1469-big-Statistics.db,

Version-Release number of selected component (if applicable):
3.4.1

UNCONFIRMED

Actual results:

..8<
INFO  20:28:24 JVM Arguments: [-Duser.home=/home/jboss, -Duser.name=jboss, -Dcassandra.commitlog.ignorereplayerrors=true, -ea, -javaagent:/opt/apache-cassandra/lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms1955M, -Xmx1955M, -Xmn400M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=/opt/apache-cassandra/conf/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:/opt/apache-cassandra/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=/opt/apache-cassandra/lib/sigar-bin, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/apache-cassandra/logs, -Dcassandra.storagedir=/opt/apache-cassandra/data, -Dcassandra-foreground=yes]
WARN  20:28:24 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  20:28:24 jemalloc shared library could not be preloaded to speed up memory allocations
WARN  20:28:24 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN  20:28:24 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  20:28:24 Initializing SIGAR library
WARN  20:28:24 Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true 
WARN  20:28:24 Directory /opt/apache-cassandra/data/saved_caches doesn't exist
ERROR 20:28:24 Detected unreadable sstables /cassandra_data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-1469-big-Statistics.db,

Providing full logs in private comments

Comment 5 Steven Walter 2017-06-17 22:32:04 UTC
Determined images from a different OCP version were in use. Closing bug pending fixing the image tags.