Description of problem: It will meet syntax errors in hawkular-cassandra pod events, like below: 1h 1h 1 {kubelet openshift-138.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Warning Unhealthy Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected Cassandra not in the up and normal state. Current state is Version-Release number of selected component (if applicable): openshift v3.2.0.8 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 openshift3/metrics-deployer d3b5bd02c6ad openshift3/metrics-hawkular-metrics 0d825e62d05a openshift3/metrics-heapster 9a6aa3a55a44 openshift3/metrics-cassandra 2f9af4d01e97 How reproducible: always Steps to Reproduce: 1. Deploy the metrics stack 2. Describe hawkular-cassandra pod after running oc describe pod hawkular-cassandra-1-ymcjs Actual results: <-----------snip--------------> 1h 1h 1 {kubelet openshift-138.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Normal Created Created container with docker id 9b4807db0e1e 1h 1h 1 {kubelet openshift-138.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Warning Unhealthy Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'. /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected Cassandra not in the up and normal state. Current state is 1h 1h 1 {kubelet openshift-138.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Warning Unhealthy Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'. Cassandra not in the up and normal state. Current state is /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected 1h 1h 1 {kubelet openshift-138.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Warning Unhealthy Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory /opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: unary operator expected Cassandra not in the up and normal state. Current state is Expected results: Should not meet syntax errors in pod events. Additional info:
Ok, I can confirm, the container does start up properly and there is no restarts, but there are some error messages being logged in events which need to be cleaned up.
other not friendly error messages: 8m 8m 1 {kubelet openshift-136.lab.sjc.redhat.com} spec.containers{hawkular-metrics} Warning Unhealthy Liveness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>. Traceback (most recent call last): File "/opt/hawkular/scripts/hawkular-metrics-liveness.py", line 44, in <module> if int(uptime) < 120: ValueError: invalid literal for int() with base 10: ''
This should be fixed for 3.3
It's fixed, checked with the metrics 3.3 images, the testing result as below: [chunchen@F17-CCY daily]$ oc describe pod hawkular-cassandra-1-iq1e5 <-------------------sinp-----------------> Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 8m 8m 1 {default-scheduler } Normal Scheduled Successfully assigned hawkular-cassandra-1-iq1e5 to openshift-111.lab.sjc.redhat.com 8m 8m 1 {kubelet openshift-111.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Normal Pulling pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/metrics-cassandra:3.3.0" 5m 5m 1 {kubelet openshift-111.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Normal Pulled Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/metrics-cassandra:3.3.0" 5m 5m 1 {kubelet openshift-111.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Normal Created Created container with docker id 557468c9c78a 5m 5m 1 {kubelet openshift-111.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Normal Started Started container with docker id 557468c9c78a 5m 5m 2 {kubelet openshift-111.lab.sjc.redhat.com} spec.containers{hawkular-cassandra-1} Warning Unhealthy Readiness probe failed:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1933
I have this error in a v3.2 production cluster. This bug says it is solved ... for v3.3. Could you push the fix for v3.2?
All that the fix does here is to remove a part of the error message that shouldn't be displayed. And it only affects Cassandra's event log when its first started. Are you experiencing any functional problems? Or only concerned about that extra part of the error message?
Effectively, the issue was somewhere else. This was only a little bit misleading.