Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1348824

Summary: [docker1.10] hawkular-cassandra in CrashLoopBackOff against ose-3.1 with docker-1.10
Product: OpenShift Container Platform Reporter: chunchen <chunchen>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED WONTFIX QA Contact: Peng Li <penli>
Severity: low Docs Contact:
Priority: medium    
Version: 3.1.0CC: aos-bugs, bleanhar, wsun
Target Milestone: ---   
Target Release: 3.2.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-31 16:14:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
hawkular-cassandra logs
none
hawkular-cassandra events none

Description chunchen 2016-06-22 07:41:56 UTC 
Description of problem:
While deploying the metrics, the hawkular-cassandra goes to crash loop backoff state.

The metrics work well against ose-3.1 with docker-1.8(docker-1.8.2-10.el7.x86_64)

Version-Release number of selected component (if applicable):
openshift v3.1.1.6-64-g80b61da
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.2
Docker 1.10.3(docker-common-1.10.3-44.el7.x86_64)

brew-pulp-docker01...com:8888/openshift3/metrics-hawkular-metrics   3.1.1               d4ec499660ac
brew-pulp-docker01...com:8888/openshift3/metrics-heapster           3.1.1               ad15738d82ec
brew-pulp-docker01...com:8888/openshift3/metrics-deployer           3.1.1               684da28d42f4
brew-pulp-docker01...com:8888/openshift3/metrics-cassandra          3.1.1               b8b230cdea46

How reproducible:
Always

Steps to Reproduce:
1. Log into OpenShift and create a project named "chunchen"
oc new-project chunchen

2. Deploy metrics stack
oc create serviceaccount metrics-deployer

oadm policy add-cluster-role-to-user cluster-reader system:serviceaccount:chunchen:heapster'

oc policy add-role-to-user edit system:serviceaccount:chunchen:metrics-deployer

oc secrets new metrics-deployer nothing=/dev/null

oc new-app metrics-deployer-template -p IMAGE_PREFIX=brew-pulp-docker01...com:8888/openshift3/,IMAGE_VERSION=3.1.1,MASTER_URL=https://$MASTER:8443,HAWKULAR_METRICS_HOSTNAME=hawkular-metrics.0621-vwc.qe.rhcloud.com,USE_PERSISTENT_STORAGE=false

3. Check the metrics pods
oc get pods

Actual results:
[root@openshift-227 ~]# oc get pod -n chunchen
NAME                         READY     STATUS             RESTARTS   AGE
hawkular-cassandra-1-rfzjy   0/1       CrashLoopBackOff   24         1h
hawkular-metrics-o969v       0/1       CrashLoopBackOff   26         1h
heapster-qm9gn               0/1       CrashLoopBackOff   27         1h
metrics-deployer-2zibu       0/1       Completed          0          1h

Expected results:
Should deploy metrics successfully.

Additional info:
The docker logs for hawkular-cassandra containers:

[root@openshift-231 ~]# docker logs 4ff27770e789
cat: /etc/ld.so.conf.d/*.conf: No such file or directory
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
INFO  22:48:36 Loading settings from file:/opt/apache-cassandra-2.2.1.redhat-2/conf/cassandra.yaml
INFO  22:48:36 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-rfzjy; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-rfzjy; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.hawkular.openshift.cassandra.OpenshiftSeedProvider, parameters=[{seeds=hawkular-cassandra-1-rfzjy}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  22:48:36 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  22:48:36 Global memtable on-heap threshold is enabled at 125MB
INFO  22:48:36 Global memtable off-heap threshold is enabled at 125MB
WARN  22:48:38 UnknownHostException for service 'hawkular-cassandra-nodes'. It may not be up yet. Trying again
WARN  22:48:40 UnknownHostException for service 'hawkular-cassandra-nodes'. It may not be up yet. Trying again
WARN  22:48:42 UnknownHostException for service 'hawkular-cassandra-nodes'. It may not be up yet. Trying again


[root@openshift-231 ~]# docker logs 6b9a61d905d4
cat: /etc/ld.so.conf.d/*.conf: No such file or directory
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
Comment 1 Brenton Leanhardt 2016-06-22 08:19:23 UTC 
Lowering the severity simply because we haven't decided if this is even supported.
Comment 2 Matt Wringe 2016-06-22 13:17:06 UTC 
Can you please provide the logs for the Cassandra pod which was killed? You can normally get this with 'oc logs -p' which gives the logs of the previous pod.

Attaching the logs as a file is usually easier to parse than pasting it in the comment section.

Also, what is under the events tab for this Cassandra pod?
Comment 3 chunchen 2016-06-23 03:24:00 UTC 
Created attachment 1171228 [details]
hawkular-cassandra logs
Comment 4 chunchen 2016-06-23 03:24:39 UTC 
Created attachment 1171229 [details]
hawkular-cassandra events
Comment 5 Matt Wringe 2016-06-23 18:22:50 UTC 
If those are the full logs using 'oc logs -p', then Cassandra appears to be starting up. So its not that Cassandra is failing to start.

From the events, it looks like something is killing the Cassandra container, which would most likely be something like a lifecycle script. Except in Cassandra 3.1.1 we don't have a lifecycle script which would kill the pod, it should just prevent the pod from reaching the ready state.

Is the only main difference now that you are running on Docker 1.10 instead of 1.8?
Comment 6 chunchen 2016-06-24 06:52:16 UTC 
Yes, the main difference is docker version, docker 1.10 is running in one environment and docker 1.8 is running in another one.
Comment 9 Matt Wringe 2016-10-31 16:14:26 UTC 
Closing this as WONTFIX as it appears this is not a supported configuration.