Description of problem: OCP 3.4 ships with Cassandra 3.0.9 which uses Netty 4.0.23. There is a severe bug in 4.0.23, https://github.com/netty/netty/issues/3057, which can lead to OutOfMemoryErrors. This issue was tracked in Cassandra under https://issues.apache.org/jira/browse/CASSANDRA-13114 and https://issues.apache.org/jira/browse/CASSANDRA-13126. We should upgrade to the latest 3.0.x release of Cassandra which is currently 3.0.13. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Do we want to move to 3.0.13? or to 3.0.11 (I think) which we are already using in 3.6?
3.0.13(In reply to Matt Wringe from comment #1) > Do we want to move to 3.0.13? or to 3.0.11 (I think) which we are already > using in 3.6? 3.0.13
There is a bad bug in 3.0.13 - https://issues.apache.org/jira/browse/CASSANDRA-13559. We will want to go with 3.0.12 or 3.0.14 if it is released soon.
Same issue with https://bugzilla.redhat.com/show_bug.cgi?id=1457501, will confirm with developers whether it could closed if memory leaks issues were not found.
# oc rsh ${hawkular-cassandra-pod} sh-4.2$ env | grep -i ver JBOSS_IMAGE_VERSION=1.3 JAVA_VERSION=1.8.0 CASSANDRA_VERSION=3.0.12.redhat-1 CASSANDRA_VERSION=3.0.12.redhat-1, and did sanity testing, metrics worked well. Since the memory leak is documented in the upstream Netty ticket, so moving to the fix version is sufficient. Please change to ON_QA then I can close it, if you want to trigger the errata process, MODIFIED status is OK for me. # openshift version openshift v3.4.0.39 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 Image id from brew registry # docker images | grep metrics metrics-hawkular-metrics 3.4.1 159bf3c1aff0 33 hours ago 1.262 GB metrics-deployer 3.4.1 8ea4546c0ed5 4 days ago 893 MB metrics-cassandra 3.4.1 a8a1940f570e 6 days ago 545.1 MB metrics-heapster 3.4.1 93fd4e9bb041 6 days ago 318 MB
CASSANDRA_VERSION is 3.0.12.redhat-1, did sanity testing, no issue was found # oc rsh ${hawkular-cassandra-pod} sh-4.2$ env | grep -i ver JBOSS_IMAGE_VERSION=1.3 JAVA_VERSION=1.8.0 CASSANDRA_VERSION=3.0.12.redhat-1 Testing env: # openshift version openshift v3.4.1.42 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 Imags from brew registry # docker images | grep metrics metrics-deployer 3.4.1 bc7efb9c7533 34 hours ago 864.2 MB metrics-hawkular-metrics 3.4.1 4b7a8114d05c 34 hours ago 1.262 GB metrics-cassandra 3.4.1 8bb36cbbb82d 34 hours ago 572.1 MB metrics-heapster 3.4.1 d0cd20203416 34 hours ago 318 MB
Set it to VERIFIED based on Comment 11
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1640
*** Bug 1464932 has been marked as a duplicate of this bug. ***