Red Hat Bugzilla – Bug 1494673
Cassandra readiness probe can incorrectly fail in multi node setup
Last modified: 2017-11-28 17:12:28 EST
Our Cassandra readiness probe will parse the output of 'nodetool status' to determine if the Cassandra instance is in the 'up' and 'normal' state. Our string parsing of the output can have an issue in certain situations. If the string value of the current host's ip address is contained within the ip of another node in the cluster, then we will try and parse two lines of the output instead of just one. For instance, consider the case where we have two nodes in our Cassandra cluster where their ip addresses are '172.17.0.3' and '172.17.0.3' ('72.17.0.3' and '172.17.0.3' would also cause a problem as well). How we are parsing this output, our script would incorrectly try and handle both entries from 'nodetool status' instead of just the one. This will cause the readiness probe to get unexpected information and fail. If the pod is brought down and restarted, it should be granted a new ip address which should not conflict with the second ip address anymore and then be able to continue.
Simple PR which fixes this issue by checking for whitespace before and after the ip address, thus preventing the script from considering the ip address the same: https://github.com/openshift/origin-metrics/pull/380
@Matt Which image version contain the fix? Do we still need to verify it failed with previous versions?
Tested with currently latest image:metrics-cassandra-v3.7.0-0.135.0.0, it returned "Cassandra is in the up and normal state"
(In reply to Junqi Zhao from comment #2) > @Matt > Which image version contain the fix? The latest 3.7 release should have this fixed. > Do we still need to verify it failed with previous versions? Its the exact same change as https://bugzilla.redhat.com/show_bug.cgi?id=1496228 which I believe was verified there.
Closed based on Comment 3 and Comment 4
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188