Bug 1462461 - Cassandra in CrashLoopBackOff with sstables error message
Cassandra in CrashLoopBackOff with sstables error message
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metrics (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Matt Wringe
Junqi Zhao
Depends On:
  Show dependency treegraph
Reported: 2017-06-17 16:52 EDT by Steven Walter
Modified: 2017-06-17 18:32 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-06-17 18:32:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Steven Walter 2017-06-17 16:52:39 EDT
Description of problem:
Cassandra pod in CrashLoopBackOff. Cassandra logs has message like:

ERROR 20:28:24 Detected unreadable sstables /cassandra_data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-1469-big-Statistics.db,

Version-Release number of selected component (if applicable):


Actual results:

INFO  20:28:24 JVM Arguments: [-Duser.home=/home/jboss, -Duser.name=jboss, -Dcassandra.commitlog.ignorereplayerrors=true, -ea, -javaagent:/opt/apache-cassandra/lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms1955M, -Xmx1955M, -Xmn400M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=/opt/apache-cassandra/conf/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:/opt/apache-cassandra/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=/opt/apache-cassandra/lib/sigar-bin, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/apache-cassandra/logs, -Dcassandra.storagedir=/opt/apache-cassandra/data, -Dcassandra-foreground=yes]
WARN  20:28:24 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  20:28:24 jemalloc shared library could not be preloaded to speed up memory allocations
WARN  20:28:24 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN  20:28:24 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  20:28:24 Initializing SIGAR library
WARN  20:28:24 Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true 
WARN  20:28:24 Directory /opt/apache-cassandra/data/saved_caches doesn't exist
ERROR 20:28:24 Detected unreadable sstables /cassandra_data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-1469-big-Statistics.db,

Providing full logs in private comments
Comment 5 Steven Walter 2017-06-17 18:32:04 EDT
Determined images from a different OCP version were in use. Closing bug pending fixing the image tags.

Note You need to log in before you can comment on or make changes to this bug.