Bug 1322275

Summary: Met syntax errors "/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected"
Product: OpenShift Container Platform Reporter: chunchen <chunchen>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED ERRATA QA Contact: chunchen <chunchen>
Severity: low Docs Contact:
Priority: medium    
Version: 3.2.0CC: aos-bugs, boris.ruppert, jlee, marc.jadoul, nbhatt, tdawson, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-27 09:37:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description chunchen 2016-03-30 07:56:14 UTC
Description of problem:
It will meet syntax errors in hawkular-cassandra pod events, like below:

1h	1h	1	{kubelet openshift-138.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected
Cassandra not in the up and normal state. Current state is

Version-Release number of selected component (if applicable):
openshift v3.2.0.8
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

openshift3/metrics-deployer d3b5bd02c6ad
openshift3/metrics-hawkular-metrics 0d825e62d05a
openshift3/metrics-heapster 9a6aa3a55a44
openshift3/metrics-cassandra 2f9af4d01e97

How reproducible:
always

Steps to Reproduce:
1. Deploy the metrics stack
2. Describe hawkular-cassandra pod after running
oc describe pod hawkular-cassandra-1-ymcjs

Actual results:
<-----------snip-------------->
  1h		1h		1	{kubelet openshift-138.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Normal		Created		Created container with docker id 9b4807db0e1e
  1h		1h		1	{kubelet openshift-138.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected
Cassandra not in the up and normal state. Current state is 

  1h	1h	1	{kubelet openshift-138.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.
Cassandra not in the up and normal state. Current state is 
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected

  1h	1h	1	{kubelet openshift-138.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Warning	Unhealthy	Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory
/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected
Cassandra not in the up and normal state. Current state is

Expected results:
Should not meet syntax errors in pod events.

Additional info:

Comment 1 Matt Wringe 2016-03-30 14:11:50 UTC
Ok, I can confirm, the container does start up properly and there is no restarts, but there are some error messages being logged in events which need to be cleaned up.

Comment 2 chunchen 2016-06-24 06:56:42 UTC
other not friendly error messages:

  8m		8m		1	{kubelet openshift-136.lab.sjc.redhat.com}	spec.containers{hawkular-metrics}	Warning		Unhealthy	Liveness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>.
Traceback (most recent call last):
  File "/opt/hawkular/scripts/hawkular-metrics-liveness.py", line 44, in <module>
    if int(uptime) < 120:
ValueError: invalid literal for int() with base 10: ''

Comment 3 Matt Wringe 2016-08-05 14:34:50 UTC
This should be fixed for 3.3

Comment 4 chunchen 2016-08-08 07:34:19 UTC
It's fixed, checked with the metrics 3.3 images, the testing result as below:

[chunchen@F17-CCY daily]$ oc describe pod hawkular-cassandra-1-iq1e5
<-------------------sinp----------------->
Events:
  FirstSeen	LastSeen	Count	From						SubobjectPath				Type		Reason		Message
  ---------	--------	-----	----						-------------				--------	------		-------
  8m		8m		1	{default-scheduler }									Normal		Scheduled	Successfully assigned hawkular-cassandra-1-iq1e5 to openshift-111.lab.sjc.redhat.com
  8m		8m		1	{kubelet openshift-111.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Normal		Pulling		pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/metrics-cassandra:3.3.0"
  5m		5m		1	{kubelet openshift-111.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Normal		Pulled		Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/metrics-cassandra:3.3.0"
  5m		5m		1	{kubelet openshift-111.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Normal		Created		Created container with docker id 557468c9c78a
  5m		5m		1	{kubelet openshift-111.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Normal		Started		Started container with docker id 557468c9c78a
  5m		5m		2	{kubelet openshift-111.lab.sjc.redhat.com}	spec.containers{hawkular-cassandra-1}	Warning		Unhealthy	Readiness probe failed:

Comment 10 errata-xmlrpc 2016-09-27 09:37:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933

Comment 11 Marc Jadoul 2017-05-22 13:52:23 UTC
I have this error in a v3.2 production cluster. This bug says it is solved ... for v3.3. 

Could you push the fix for v3.2?

Comment 12 Matt Wringe 2017-05-23 20:38:49 UTC
All that the fix does here is to remove a part of the error message that shouldn't be displayed. And it only affects Cassandra's event log when its first started.

Are you experiencing any functional problems? Or only concerned about that extra part of the error message?

Comment 13 Marc Jadoul 2017-07-18 12:49:13 UTC
Effectively, the issue was somewhere else. This was only a little bit misleading.