Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1607984

Summary:	[3.10][CRI-O] Cassandra doesn't start on crio
Product:	OpenShift Container Platform	Reporter:	Junqi Zhao <juzhao>
Component:	Hawkular	Assignee:	Antonio Murdaca <amurdaca>
Status:	CLOSED WORKSFORME	QA Contact:	Junqi Zhao <juzhao>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.10.0	CC:	amurdaca, anli, aos-bugs, jhonce, jokerman, jsanda, miburman, mmccomas, mwringe, trankin, wmeng
Target Milestone:	---	Keywords:	Regression
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1517652	Environment:
Last Closed:	2018-07-25 06:43:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 1 Junqi Zhao 2018-07-24 16:39:41 UTC

The latest image with cri-o 1.10.5 is also have this issue, should seek help from cri-o team

metrics-cassandra-v3.10.23-1
metrics-hawkular-metrics-v3.10.23-1
metrics-schema-installer-v3.10.23-1
metrics-heapster-v3.10.23-1

Comment 2 Junqi Zhao 2018-07-25 02:16:35 UTC

Blocks metrics installation

Comment 3 Anping Li 2018-07-25 03:35:16 UTC

Both crictl exec and runc exec works well.

# crictl exec  7140ef4831fe7ced8d1f480ba3046f2f3dff8e85436dad015f7558821c882549 /opt/apache-cassandra/bin/cassandra-docker-ready.sh
W0724 23:33:33.423292   29209 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock".
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Cassandra is in the up and normal state. It is now ready.


#runc exec 7140ef4831fe7ced8d1f480ba3046f2f3dff8e85436dad015f7558821c882549 /opt/apache-cassandra/bin/cassandra-docker-ready.sh
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Cassandra is in the up and normal state. It is now ready

Comment 4 Anping Li 2018-07-25 03:41:40 UTC

The interesting thing is that when the hawkular-cassandra pod is deleted, the new  pod could be ready.
so the workaround is to delete the hawkular-cassandra pod.

Comment 5 Junqi Zhao 2018-07-25 06:04:30 UTC

the default timeoutSeconds is 1 second, but the readiness check takes more than 1 second to get the response

# time oc rsh pod/hawkular-cassandra-1-prh6t /opt/apache-cassandra/bin/cassandra-docker-ready.sh
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Cassandra is in the up and normal state. It is now ready.

real	0m1.360s
user	0m0.140s
sys	0m0.041s


after changing it to bigger value in roles/openshift_metrics/templates/hawkular_cassandra_rc.j2, added timeoutSeconds: 10, metrics works well.

        readinessProbe:
          exec:
            command:
            - "/opt/apache-cassandra/bin/cassandra-docker-ready.sh"
          timeoutSeconds: 10

Comment 6 Junqi Zhao 2018-07-25 06:43:32 UTC

Per Comment 5, close this defect and filed Bug 1608216 to change the default timeoutSeconds for hawkular-cassandra pod's readiness check