|Summary:||Cassandra Java heap parameters can configured incorrectly|
|Product:||OpenShift Container Platform||Reporter:||John Sanda <jsanda>|
|Component:||Hawkular||Assignee:||Matt Wringe <mwringe>|
|Status:||CLOSED ERRATA||QA Contact:||Junqi Zhao <juzhao>|
|Version:||3.6.0||CC:||aos-bugs, jsanda, pdwyer, pweil, snegrea|
|Fixed In Version:||3.7.0||Doc Type:||If docs needed, set a value|
|Doc Text:||Story Points:||---|
|Last Closed:||2017-11-28 22:01:28 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
Description John Sanda 2017-07-14 19:15:33 UTC
Description of problem: We pass the -Xms, -Xmx, and -Xmn flags to the Cassandra JVM. -Xms sets the minimum size for all of the heap. -Xmx set the max size for all of the heap. -Xmn sets the size of the new generation. The heap is basically divided into two sections, the new or young generation and the old generation. We calculate the value for -Xmn based on the number of cpu cores. There are times, like when there are no cpu limits, that we can end up with -Xmn having a value larger than -Xmx. If that happens, the JVM will log a warning at start up like this: OpenJDK 64-Bit Server VM warning: MaxNewSize (4096000k) is equal to or greater than the entire heap (1048576k). A new max generation size of 1048512k will be used. We need to fix this. The JVM will dynamically resize the generations as needed, and this can also be controlled with other flags which we do not set. For the type of work loads with which we are dealing, I think we generally want the new generation to be between 1/4 and 1/2 of the total heap. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Comment 2 Matt Wringe 2017-07-18 13:33:08 UTC
We have been setting it this way based on the Cassandra recommendations (although we were not taking into account the total memory available). How serious is this issue?
Comment 3 John Sanda 2017-07-19 22:08:02 UTC
I am not sure how serious it is. We know it does not prevent Cassandra from starting, and I only think it is an issue where there is unlimited cpu. Based on some additional research I think we should set the new generation to 1/4 of the total heap. I think that is a reasonable default.
Comment 4 Matt Wringe 2017-09-29 18:39:59 UTC
We are now setting the heap newsize to be 1/3 of the heapsize.
Comment 5 Junqi Zhao 2017-10-09 03:06:01 UTC
Tested with currently latest metrics-cassandra image:metrics-cassandra:v3.7.0-0.144.0.0 and set openshift_metrics_cassandra_limits_memory=3Gi From metrics-cassandra logs, HEAP_NEWSIZE is 1/3 of MAX_HEAP_SIZE. # oc logs hawkular-cassandra-1-2lzzf The MAX_HEAP_SIZE envar is not set. Basing the MAX_HEAP_SIZE on the available memory limit for the pod (3221225472). The memory limit is between 2 and 4GB. Setting max_heap_size to 1GB. The MAX_HEAP_SIZE has been set to 1024M The HEAP_NEWSIZE envar is not set. Setting the HEAP_NEWSIZE to one third the MAX_HEAP_SIZE: 341M Also tested the default limit memory size(2G), from pod log, the HEAP_NEWSIZE is also 1/3 of MAX_HEAP_SIZE now. # oc logs hawkular-cassandra-1-h2f6j The MAX_HEAP_SIZE envar is not set. Basing the MAX_HEAP_SIZE on the available memory limit for the pod (2000000000). The memory limit is less than 2GB. Using 1/2 of available memory for the max_heap_size. The MAX_HEAP_SIZE has been set to 953M The HEAP_NEWSIZE envar is not set. Setting the HEAP_NEWSIZE to one third the MAX_HEAP_SIZE: 317M Close it as VERIFIED
Comment 8 errata-xmlrpc 2017-11-28 22:01:28 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188