Attachment 1455336 Details for Bug 1596261 – Logs for crashing pod

Logs for crashing pod

file_1596261.txt (text/plain), 29.94 KB, created by Justin Pierce on 2018-06-28 13:37:07 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Justin Pierce

Created: 2018-06-28 13:37:07 UTC

Size: 29.94 KB

patch

obsolete

>[root@free-stg-master-03fb6 ~]# oc get pod hawkular-cassandra-1-jtlpx -o=json | grep image
>...
>"image": "registry.reg-aws.openshift.com:443/openshift3/metrics-cassandra:v3.10.9",
>
>
>[root@free-stg-master-03fb6 ~]# oc project openshift-infra
>oc get pods
>Now using project "openshift-infra" on server "https://172.31.78.254:443".
>[root@free-stg-master-03fb6 ~]# oc get pods
>NAME                             READY     STATUS             RESTARTS   AGE
>bootstrap-autoapprover-0         1/1       Running            0          17h
>hawkular-cassandra-1-jtlpx       0/1       CrashLoopBackOff   197        17h
>hawkular-cassandra-2-mz6qx       0/1       CrashLoopBackOff   171        17h
>hawkular-cassandra-3-p7bf5       0/1       CrashLoopBackOff   171        17h
>hawkular-metrics-ld8xp           0/1       Running            110        17h
>hawkular-metrics-schema-7ggfs    1/1       Running            0          17h
>heapster-jlb2f                   0/1       Running            110        17h
>hibernation-10-build             0/1       Completed          0          84d
>hibernation-53-8t2bp             1/1       Running            0          17h
>oso-console-extensions-6-wspd5   1/1       Running            2          1d
>user-analytics-1-dqb8h           1/1       Running            1          1d
>[root@free-stg-master-03fb6 ~]# oc logs hawkular-cassandra-1-jtlpx
>The MAX_HEAP_SIZE envar is not set. Basing the MAX_HEAP_SIZE on the available memory limit for the pod (4294967296).
>The memory limit is between 2 and 4GB. Setting max_heap_size to 1GB.
>The MAX_HEAP_SIZE has been set to 1024M
>The HEAP_NEWSIZE envar is not set. Setting the HEAP_NEWSIZE to one third the MAX_HEAP_SIZE: 341M
>About to generate seeds
>Trying to access the Seed list [try #1]
>Trying to access the Seed list [try #2]
>Trying to access the Seed list [try #3]
>Setting seeds to be hawkular-cassandra-1-jtlpx
>Creating the Cassandra keystore from the Secret's cert data
>Converting the PKCS12 keystore into a Java Keystore
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>Importing keystore /opt/apache-cassandra/conf/cassandra.pkcs12 to /opt/apache-cassandra/conf/.keystore...
>Entry for alias cassandra successfully imported.
>Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
>[Storing /opt/apache-cassandra/conf/.keystore]
>
>Warning:
>The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /opt/apache-cassandra/conf/.keystore -destkeystore /opt/apache-cassandra/conf/.keystore -deststoretype pkcs12".
>Building the trust store for inter node communication
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>Certificate was added to keystore
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>Certificate was added to keystore
>/opt/apache-cassandra/bin/cassandra-docker.sh: line 308: cd: /home/jboss: Permission denied
>Building the trust store for client communication
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>Certificate was added to keystore
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>Certificate was added to keystore
>Generating self signed certificates for the local client for cqlsh
>Generating a 4096 bit RSA private key
>.....++
>.++
>writing new private key to '.cassandra.local.client.key'
>-----
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>Certificate was added to keystore
>The previous version of Cassandra was 3.0.15.redhat-1. The current version is 3.0.15.redhat-1
>cat: /etc/ld.so.conf.d/*.conf: No such file or directory
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>OpenJDK 64-Bit Server VM warning: Cannot open file /opt/apache-cassandra/logs/gc.log due to No such file or directory
>
>CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
>CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
>CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
>CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
>CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
>CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
>CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
>CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
>CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
>CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
>CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
>CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
>CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
>CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
>CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
>CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
>CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
>CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
>CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
>CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
>CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
>INFO  [main] 2018-06-28 13:32:42,490 YamlConfigurationLoader.java:85 - Configuration location: file:/opt/apache-cassandra-3.0.15.redhat-1/conf/cassandra.yaml
>INFO  [main] 2018-06-28 13:32:42,618 Config.java:457 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=false; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@3ba9ad43; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=512; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-jtlpx; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=null; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-jtlpx; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=hawkular-cassandra-1-jtlpx}; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=30; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
>INFO  [main] 2018-06-28 13:32:42,619 DatabaseDescriptor.java:325 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
>INFO  [main] 2018-06-28 13:32:42,739 DatabaseDescriptor.java:432 - Global memtable on-heap threshold is enabled at 247MB
>INFO  [main] 2018-06-28 13:32:42,740 DatabaseDescriptor.java:436 - Global memtable off-heap threshold is enabled at 247MB
>INFO  [main] 2018-06-28 13:32:42,885 CassandraDaemon.java:434 - Hostname: hawkular-cassandra-1-jtlpx
>INFO  [main] 2018-06-28 13:32:42,885 CassandraDaemon.java:441 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_171
>INFO  [main] 2018-06-28 13:32:42,886 CassandraDaemon.java:442 - Heap size: 1038024704/1038024704
>INFO  [main] 2018-06-28 13:32:42,886 CassandraDaemon.java:445 - Code Cache Non-heap memory: init = 2555904(2496K) used = 3798720(3709K) committed = 3866624(3776K) max = 251658240(245760K)
>INFO  [main] 2018-06-28 13:32:42,887 CassandraDaemon.java:445 - Metaspace Non-heap memory: init = 0(0K) used = 17835248(17417K) committed = 18219008(17792K) max = -1(-1K)
>INFO  [main] 2018-06-28 13:32:42,887 CassandraDaemon.java:445 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2086360(2037K) committed = 2228224(2176K) max = 1073741824(1048576K)
>INFO  [main] 2018-06-28 13:32:42,887 CassandraDaemon.java:445 - Par Eden Space Heap memory: init = 286130176(279424K) used = 131683728(128597K) committed = 286130176(279424K) max = 286130176(279424K)
>INFO  [main] 2018-06-28 13:32:42,887 CassandraDaemon.java:445 - Par Survivor Space Heap memory: init = 35717120(34880K) used = 0(0K) committed = 35717120(34880K) max = 35717120(34880K)
>INFO  [main] 2018-06-28 13:32:42,888 CassandraDaemon.java:445 - CMS Old Gen Heap memory: init = 716177408(699392K) used = 0(0K) committed = 716177408(699392K) max = 716177408(699392K)
>INFO  [main] 2018-06-28 13:32:42,888 CassandraDaemon.java:447 - Classpath: /opt/apache-cassandra/conf:/opt/apache-cassandra/build/classes/main:/opt/apache-cassandra/build/classes/thrift:/opt/apache-cassandra/lib/ST4-4.0.8.jar:/opt/apache-cassandra/lib/airline-0.6.jar:/opt/apache-cassandra/lib/ant-junit-1.9.4.jar:/opt/apache-cassandra/lib/antlr-runtime-3.5.2.jar:/opt/apache-cassandra/lib/apache-cassandra-3.0.15.redhat-1.jar:/opt/apache-cassandra/lib/apache-cassandra-clientutil-3.0.15.redhat-1.jar:/opt/apache-cassandra/lib/apache-cassandra-thrift-3.0.15.redhat-1.jar:/opt/apache-cassandra/lib/asm-5.0.4.jar:/opt/apache-cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/apache-cassandra/lib/commons-cli-1.1.jar:/opt/apache-cassandra/lib/commons-codec-1.2.jar:/opt/apache-cassandra/lib/commons-lang3-3.1.jar:/opt/apache-cassandra/lib/commons-math3-3.2.jar:/opt/apache-cassandra/lib/compress-lzf-0.8.4.jar:/opt/apache-cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/apache-cassandra/lib/disruptor-3.0.1.jar:/opt/apache-cassandra/lib/ecj-4.4.2.jar:/opt/apache-cassandra/lib/guava-18.0.jar:/opt/apache-cassandra/lib/high-scale-lib-1.0.6.jar:/opt/apache-cassandra/lib/jackson-core-asl-1.9.2.jar:/opt/apache-cassandra/lib/jackson-mapper-asl-1.9.2.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar:/opt/apache-cassandra/lib/javax.inject.jar:/opt/apache-cassandra/lib/jbcrypt-0.3m.jar:/opt/apache-cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/jmx_prometheus_javaagent.jar:/opt/apache-cassandra/lib/jna-4.2.2.jar:/opt/apache-cassandra/lib/joda-time-2.4.jar:/opt/apache-cassandra/lib/json-simple-1.1.jar:/opt/apache-cassandra/lib/jstackjunit-0.0.1.jar:/opt/apache-cassandra/lib/libthrift-0.9.2.jar:/opt/apache-cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/logback-classic-1.1.3.jar:/opt/apache-cassandra/lib/logback-core-1.1.3.jar:/opt/apache-cassandra/lib/lz4-1.3.0.jar:/opt/apache-cassandra/lib/metrics-core-3.1.0.jar:/opt/apache-cassandra/lib/metrics-jvm-3.1.0.jar:/opt/apache-cassandra/lib/metrics-logback-3.1.0.jar:/opt/apache-cassandra/lib/netty-all-4.0.44.Final.jar:/opt/apache-cassandra/lib/ohc-core-0.4.3.jar:/opt/apache-cassandra/lib/ohc-core-j8-0.4.3.jar:/opt/apache-cassandra/lib/reporter-config-base-3.0.0.jar:/opt/apache-cassandra/lib/reporter-config3-3.0.0.jar:/opt/apache-cassandra/lib/sigar-1.6.4.jar:/opt/apache-cassandra/lib/slf4j-api-1.7.7.jar:/opt/apache-cassandra/lib/snakeyaml-1.11.jar:/opt/apache-cassandra/lib/snappy-java-1.1.1.7.jar:/opt/apache-cassandra/lib/stream-2.5.2.jar:/opt/apache-cassandra/lib/thrift-server-0.3.7.jar:/opt/apache-cassandra/lib/jsr223/*/*.jar:/opt/apache-cassandra/lib/jmx_prometheus_javaagent.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar
>INFO  [main] 2018-06-28 13:32:42,889 CassandraDaemon.java:449 - JVM Arguments: [-Duser.home=/home/jboss, -Duser.name=jboss, -Dcassandra.commitlog.ignorereplayerrors=true, -javaagent:/opt/apache-cassandra/lib/jmx_prometheus_javaagent.jar=7575:/opt/hawkular/prometheus_agent/prometheus.yaml, -Xloggc:/opt/apache-cassandra/logs/gc.log, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms1024M, -Xmx1024M, -Xmn341M, -ea, -Xss256k, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:StringTableSize=1000003, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=/opt/apache-cassandra/conf/hotspot_compiler, -javaagent:/opt/apache-cassandra/lib/jamm-0.3.0.jar, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=/opt/apache-cassandra/lib/sigar-bin, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/apache-cassandra/logs, -Dcassandra.storagedir=/opt/apache-cassandra/data, -Dcassandra-foreground=yes]
>WARN  [main] 2018-06-28 13:32:42,962 NativeLibrary.java:180 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
>WARN  [main] 2018-06-28 13:32:42,962 StartupChecks.java:129 - jemalloc shared library could not be preloaded to speed up memory allocations
>WARN  [main] 2018-06-28 13:32:42,963 StartupChecks.java:161 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
>WARN  [main] 2018-06-28 13:32:42,963 StartupChecks.java:186 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
>INFO  [main] 2018-06-28 13:32:42,965 SigarLibrary.java:44 - Initializing SIGAR library
>INFO  [main] 2018-06-28 13:32:42,978 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
>WARN  [main] 2018-06-28 13:32:42,982 StartupChecks.java:254 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
>WARN  [main] 2018-06-28 13:32:43,001 StartupChecks.java:275 - Directory /opt/apache-cassandra/data/saved_caches doesn't exist
>WARN  [main] 2018-06-28 13:32:43,003 StartupChecks.java:275 - Directory /opt/apache-cassandra/data/hints doesn't exist
>INFO  [main] 2018-06-28 13:32:43,727 ColumnFamilyStore.java:391 - Initializing system.IndexInfo
>INFO  [SSTableBatchOpen:1] 2018-06-28 13:32:44,668 BufferPool.java:226 - Global buffer pool is enabled, when pool is exahusted (max is 512 mb) it will allocate on heap
>INFO  [main] 2018-06-28 13:32:44,710 CacheService.java:115 - Initializing key cache with capacity of 49 MBs.
>INFO  [main] 2018-06-28 13:32:44,718 CacheService.java:137 - Initializing row cache with capacity of 0 MBs
>INFO  [main] 2018-06-28 13:32:44,720 CacheService.java:166 - Initializing counter cache with capacity of 24 MBs
>INFO  [main] 2018-06-28 13:32:44,722 CacheService.java:177 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
>INFO  [main] 2018-06-28 13:32:44,742 ColumnFamilyStore.java:391 - Initializing system.batches
>INFO  [main] 2018-06-28 13:32:44,753 ColumnFamilyStore.java:391 - Initializing system.paxos
>INFO  [main] 2018-06-28 13:32:44,772 ColumnFamilyStore.java:391 - Initializing system.local
>INFO  [main] 2018-06-28 13:32:44,792 ColumnFamilyStore.java:391 - Initializing system.peers
>INFO  [main] 2018-06-28 13:32:44,807 ColumnFamilyStore.java:391 - Initializing system.peer_events
>INFO  [main] 2018-06-28 13:32:44,814 ColumnFamilyStore.java:391 - Initializing system.range_xfers
>INFO  [main] 2018-06-28 13:32:44,822 ColumnFamilyStore.java:391 - Initializing system.compaction_history
>INFO  [main] 2018-06-28 13:32:44,837 ColumnFamilyStore.java:391 - Initializing system.sstable_activity
>INFO  [main] 2018-06-28 13:32:44,863 ColumnFamilyStore.java:391 - Initializing system.size_estimates
>INFO  [main] 2018-06-28 13:32:44,874 ColumnFamilyStore.java:391 - Initializing system.available_ranges
>INFO  [main] 2018-06-28 13:32:44,881 ColumnFamilyStore.java:391 - Initializing system.views_builds_in_progress
>INFO  [main] 2018-06-28 13:32:44,888 ColumnFamilyStore.java:391 - Initializing system.built_views
>INFO  [main] 2018-06-28 13:32:44,894 ColumnFamilyStore.java:391 - Initializing system.hints
>INFO  [main] 2018-06-28 13:32:44,901 ColumnFamilyStore.java:391 - Initializing system.batchlog
>INFO  [main] 2018-06-28 13:32:44,907 ColumnFamilyStore.java:391 - Initializing system.schema_keyspaces
>INFO  [main] 2018-06-28 13:32:44,913 ColumnFamilyStore.java:391 - Initializing system.schema_columnfamilies
>INFO  [main] 2018-06-28 13:32:44,919 ColumnFamilyStore.java:391 - Initializing system.schema_columns
>INFO  [main] 2018-06-28 13:32:44,925 ColumnFamilyStore.java:391 - Initializing system.schema_triggers
>INFO  [main] 2018-06-28 13:32:44,933 ColumnFamilyStore.java:391 - Initializing system.schema_usertypes
>INFO  [main] 2018-06-28 13:32:44,939 ColumnFamilyStore.java:391 - Initializing system.schema_functions
>INFO  [main] 2018-06-28 13:32:44,945 ColumnFamilyStore.java:391 - Initializing system.schema_aggregates
>Exception (org.apache.cassandra.io.sstable.CorruptSSTableException) encountered during startup: Corrupted: /cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-big-Data.db
>org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-big-Data.db
>	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:218)
>	at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
>	at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:302)
>	at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:134)
>	at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1741)
>	at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:94)
>	at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:51)
>	at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:41)
>	at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:69)
>	at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:737)
>	at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:650)
>	at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:485)
>	at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:405)
>	at org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:435)
>	at org.apache.cassandra.db.SinglePartitionReadCommand$Group.executeInternal(SinglePartitionReadCommand.java:1168)
>	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:459)
>	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:446)
>	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:116)
>	at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
>	at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:906)
>	at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:362)
>	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:118)
>	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:179)
>	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
>	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
>Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-big-Data.db): corruption detected, chunk at 0 of length 47.
>	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:187)
>	... 24 more
>Caused by: java.io.IOException: Compressed lengths mismatch - 3 bytes remain
>	at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:124)
>	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:183)
>	... 24 more
>ERROR [main] 2018-06-28 13:32:45,045 CassandraDaemon.java:710 - Exception encountered during startup
>org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-big-Data.db
>	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:218) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:302) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:134) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1741) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:94) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:51) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:41) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:69) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:737) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:650) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:485) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:405) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:435) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.SinglePartitionReadCommand$Group.executeInternal(SinglePartitionReadCommand.java:1168) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:459) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:446) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:116) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:906) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:362) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:118) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:179) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/cassandra_data/data/system/local-7ad54392bcdd35a684174e047860b377/mc-1971-big-Data.db): corruption detected, chunk at 0 of length 47.
>	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:187) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	... 24 common frames omitted
>Caused by: java.io.IOException: Compressed lengths mismatch - 3 bytes remain
>	at org.apache.cassandra.io.compress.LZ4Compressor.uncompress(LZ4Compressor.java:124) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:183) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	... 24 common frames omitted

Actions: View

Attachments on bug 1596261: 1455336