Bug 1457499

Summary: Upgrade Cassandra to fix Netty memory leak
Product: OpenShift Container Platform Reporter: John Sanda <jsanda>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, gburges, jsanda, juzhao, lizhou, pdwyer, simon.gunzenreiner, smunilla, xtian
Target Milestone: ---   
Target Release: 3.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
There is a memory leak in the version of Netty that is part of Cassandra 3.0.9, the version in OCP 3.4. A version of Netty that has the fix is in Cassandra 3.0.13.
Story Points: ---
Clone Of:
: 1457501 (view as bug list) Environment:
Last Closed: 2017-07-11 10:47:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1457501    

Description John Sanda 2017-05-31 19:54:15 UTC
Description of problem:
OCP 3.4 ships with Cassandra 3.0.9 which uses Netty 4.0.23. There is a severe bug in 4.0.23, https://github.com/netty/netty/issues/3057, which can lead to OutOfMemoryErrors. This issue was tracked in Cassandra under https://issues.apache.org/jira/browse/CASSANDRA-13114 and https://issues.apache.org/jira/browse/CASSANDRA-13126. We should upgrade to the latest 3.0.x release of Cassandra which is currently 3.0.13.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Matt Wringe 2017-06-01 13:51:20 UTC
Do we want to move to 3.0.13? or to 3.0.11 (I think) which we are already using in 3.6?

Comment 2 John Sanda 2017-06-01 14:42:20 UTC
3.0.13(In reply to Matt Wringe from comment #1)
> Do we want to move to 3.0.13? or to 3.0.11 (I think) which we are already
> using in 3.6?

3.0.13

Comment 3 John Sanda 2017-06-01 22:43:46 UTC
There is a bad bug in 3.0.13 - https://issues.apache.org/jira/browse/CASSANDRA-13559. We will want to go with 3.0.12 or 3.0.14 if it is released soon.

Comment 8 Junqi Zhao 2017-06-13 09:15:53 UTC
Same issue with https://bugzilla.redhat.com/show_bug.cgi?id=1457501, will confirm with developers whether it could closed if memory leaks issues were not found.

Comment 9 Junqi Zhao 2017-06-14 02:32:17 UTC
# oc rsh ${hawkular-cassandra-pod}
sh-4.2$ env | grep -i ver
JBOSS_IMAGE_VERSION=1.3
JAVA_VERSION=1.8.0
CASSANDRA_VERSION=3.0.12.redhat-1


CASSANDRA_VERSION=3.0.12.redhat-1, and did sanity testing, metrics worked well.
Since the memory leak is documented in the upstream Netty ticket, so moving to the fix version is sufficient.

Please change to ON_QA then I can close it, if you want to trigger the errata process, MODIFIED status is OK for me.

# openshift version
openshift v3.4.0.39
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

Image id from brew registry
# docker images | grep metrics
metrics-hawkular-metrics   3.4.1               159bf3c1aff0        33 hours ago        1.262 GB
metrics-deployer           3.4.1               8ea4546c0ed5        4 days ago          893 MB
metrics-cassandra          3.4.1               a8a1940f570e        6 days ago          545.1 MB
metrics-heapster           3.4.1               93fd4e9bb041        6 days ago          318 MB

Comment 11 Junqi Zhao 2017-06-23 01:29:41 UTC
CASSANDRA_VERSION is 3.0.12.redhat-1, did sanity testing, no issue was found

# oc rsh ${hawkular-cassandra-pod}
sh-4.2$ env | grep -i ver
JBOSS_IMAGE_VERSION=1.3
JAVA_VERSION=1.8.0
CASSANDRA_VERSION=3.0.12.redhat-1

Testing env:
# openshift version
openshift v3.4.1.42
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0


Imags from brew registry
# docker images | grep metrics
metrics-deployer           3.4.1               bc7efb9c7533        34 hours ago        864.2 MB
metrics-hawkular-metrics   3.4.1               4b7a8114d05c        34 hours ago        1.262 GB
metrics-cassandra          3.4.1               8bb36cbbb82d        34 hours ago        572.1 MB
metrics-heapster           3.4.1               d0cd20203416        34 hours ago        318 MB

Comment 12 Junqi Zhao 2017-06-23 01:30:17 UTC
Set it to VERIFIED based on Comment 11

Comment 14 errata-xmlrpc 2017-07-11 10:47:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1640

Comment 15 Matt Wringe 2017-08-04 19:16:36 UTC
*** Bug 1464932 has been marked as a duplicate of this bug. ***