Bug 1188971 - ENGINE_HEAP_MAX default value as 1G must be changed
Summary: ENGINE_HEAP_MAX default value as 1G must be changed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.5.1
Assignee: Yedidyah Bar David
QA Contact: Yuri Obshansky
URL:
Whiteboard: integration
Depends On: 1185411
Blocks: 1193058 1197441
TreeView+ depends on / blocked
 
Reported: 2015-02-04 07:10 UTC by rhev-integ
Modified: 2015-04-28 18:46 UTC (History)
19 users (show)

Fixed In Version: org.ovirt.engine-root-3.5.1-1
Doc Type: Bug Fix
Doc Text:
The Manager's Java Virtual Machine heap size was configured to be 1GB by default. Large setups made the Manager run out of heap memory and required manual configuration to increase the heap size. With this update, engine-setup now automatically configures the heap size to be either a minimum of 1GB or 1/4 of the available memory. Large setups only need to use a machine with enough memory (the recommendation is 16 GB of ram), and heap size configuration will be done automatically, thus preventing out of memory failures.
Clone Of: 1185411
Environment:
Last Closed: 2015-04-28 18:46:17 UTC
oVirt Team: ---
Target Upstream Version:
ylavi: Triaged+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1190471 unspecified CLOSED [Scale] Engine runs out of memory - java.lang.OutOfMemoryError: Java heap space 2021-01-20 06:05:38 UTC
Red Hat Product Errata RHSA-2015:0888 normal SHIPPED_LIVE Moderate: Red Hat Enterprise Virtualization Manager 3.5.1 update 2015-04-28 22:40:04 UTC
oVirt gerrit 37293 master MERGED packaging: services: Update default mem heap values 2020-10-09 08:55:07 UTC
oVirt gerrit 37437 master MERGED packaging: setup: expose total memory in env 2020-10-09 08:55:07 UTC
oVirt gerrit 37438 master MERGED packaging: setup: Make engine heap size configurable 2020-10-09 08:55:07 UTC
oVirt gerrit 37501 master MERGED packaging: setup: total mem env in common 2020-10-09 08:55:07 UTC
oVirt gerrit 37575 ovirt-engine-3.5 MERGED packaging: setup: expose total memory in env 2020-10-09 08:55:07 UTC
oVirt gerrit 37576 ovirt-engine-3.5 MERGED packaging: setup: Make engine heap size configurable 2020-10-09 08:55:07 UTC
oVirt gerrit 37589 ovirt-engine-3.5 MERGED packaging: setup: pep8 2020-10-09 08:55:08 UTC

Internal Links: 1190471

Comment 2 Yuri Obshansky 2015-03-03 14:05:59 UTC
Hi Yedidyah,
I have 16 G RAM on engine server  
# cat /proc/meminfo #MemTotal: 16212064 kB
Up to bug fix description: Engine HEAP_MAX should be calculate such as
"1GB and 1/4 of available memory."
So, in my case = 1G + 4G = 5G
I see my engine process allocated with -Xms3958M 
Could you , please, clarify an algorithm?
BTW, Where it should be changed?
Cause I see old settings in file 
/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf
ENGINE_HEAP_MIN=1g
ENGINE_HEAP_MAX=1g

Comment 3 Yedidyah Bar David 2015-03-03 14:17:18 UTC
(In reply to Yuri Obshansky from comment #2)
> Hi Yedidyah,
> I have 16 G RAM on engine server  
> # cat /proc/meminfo #MemTotal: 16212064 kB
> Up to bug fix description: Engine HEAP_MAX should be calculate such as
> "1GB and 1/4 of available memory."

actually as "maximum of 1GB and 1/4 of available memory".

So for systems with <=4G, it will be 1G (as today), and for systems with >= 4G it will be 1/4 of available memory.

> So, in my case = 1G + 4G = 5G

16>4 => should be 1/4 of that, 4G

> I see my engine process allocated with -Xms3958M 

which is the result of the calculation, assuming the reported available memory was slightly less than 16G (which is normal).

> Could you , please, clarify an algorithm?
> BTW, Where it should be changed?
> Cause I see old settings in file 
> /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf
> ENGINE_HEAP_MIN=1g
> ENGINE_HEAP_MAX=1g

This it the "defaults" file.

It's written by engine-setup to /etc/ovirt-engine/engine.conf.d/10-setup-java.conf .

Comment 4 Yuri Obshansky 2015-03-03 14:19:39 UTC
I think, should be 
"maximum of 1GB OR 1/4 of available memory".

Comment 5 Yuri Obshansky 2015-03-03 14:53:20 UTC
Why -Xms3958M and not -Xms4096M?

Comment 6 Yedidyah Bar David 2015-03-03 14:57:13 UTC
(In reply to Yuri Obshansky from comment #4)
> I think, should be 
> "maximum of 1GB OR 1/4 of available memory".

Well, I am not a native English speaker. You are welcome to edit. Content Services will eventually rewrite this in better language.

(In reply to Yuri Obshansky from comment #5)
> Why -Xms3958M and not -Xms4096M?

See comment 3. The "available memory" is taken from the line "MemTotal" of /proc/meminfo .

Comment 7 Yuri Obshansky 2015-03-03 15:03:57 UTC
(In reply to Yedidyah Bar David from comment #6)
> (In reply to Yuri Obshansky from comment #4)
> > I think, should be 
> > "maximum of 1GB OR 1/4 of available memory".
> 
> Well, I am not a native English speaker. You are welcome to edit. Content
> Services will eventually rewrite this in better language.
It is OK. That the reason why I started clarify your algorithm to be sure I understand it is correct. 
> 
> (In reply to Yuri Obshansky from comment #5)
> > Why -Xms3958M and not -Xms4096M?
> 
> See comment 3. The "available memory" is taken from the line "MemTotal" of
> /proc/meminfo .
Something wrong with calculation here:
MemTotal: 16212064 kB
16212064 / 4 = 4053016
So, should be -Xms4053M, isn't it.

Comment 8 Yedidyah Bar David 2015-03-03 15:13:52 UTC
(In reply to Yuri Obshansky from comment #7)
> (In reply to Yedidyah Bar David from comment #6)
> > (In reply to Yuri Obshansky from comment #4)
> > > I think, should be 
> > > "maximum of 1GB OR 1/4 of available memory".
> > 
> > Well, I am not a native English speaker. You are welcome to edit. Content
> > Services will eventually rewrite this in better language.
> It is OK. That the reason why I started clarify your algorithm to be sure I
> understand it is correct. 

Of course.

> > 
> > (In reply to Yuri Obshansky from comment #5)
> > > Why -Xms3958M and not -Xms4096M?
> > 
> > See comment 3. The "available memory" is taken from the line "MemTotal" of
> > /proc/meminfo .
> Something wrong with calculation here:
> MemTotal: 16212064 kB
> 16212064 / 4 = 4053016
> So, should be -Xms4053M, isn't it.

16212064/1024/4 = 3958.0234375

Perhaps we better write MiB/GiB and not GB [1], to be more accurate.

[1] http://en.wikipedia.org/wiki/Binary_prefix

Comment 9 Yuri Obshansky 2015-03-03 15:25:37 UTC
(In reply to Yedidyah Bar David from comment #8)
> (In reply to Yuri Obshansky from comment #7)
> > (In reply to Yedidyah Bar David from comment #6)
> > > (In reply to Yuri Obshansky from comment #4)
> > > > I think, should be 
> > > > "maximum of 1GB OR 1/4 of available memory".
> > > 
> > > Well, I am not a native English speaker. You are welcome to edit. Content
> > > Services will eventually rewrite this in better language.
> > It is OK. That the reason why I started clarify your algorithm to be sure I
> > understand it is correct. 
> 
> Of course.
> 
> > > 
> > > (In reply to Yuri Obshansky from comment #5)
> > > > Why -Xms3958M and not -Xms4096M?
> > > 
> > > See comment 3. The "available memory" is taken from the line "MemTotal" of
> > > /proc/meminfo .
> > Something wrong with calculation here:
> > MemTotal: 16212064 kB
> > 16212064 / 4 = 4053016
> > So, should be -Xms4053M, isn't it.
> 
> 16212064/1024/4 = 3958.0234375
> 
> Perhaps we better write MiB/GiB and not GB [1], to be more accurate.
> 
> [1] http://en.wikipedia.org/wiki/Binary_prefix

Probably, you are right.
As far I know, everybody uses -Xms4096M option.
For example: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/5/html/Performance_Tuning_Guide/sect-Performance_Tuning_Guide-Java_Virtual_Machine_Tuning-Large_Page_Memory.html

Comment 10 Yedidyah Bar David 2015-03-04 07:04:39 UTC
(In reply to Yuri Obshansky from comment #9)
> As far I know, everybody uses -Xms4096M option.

Of course, because people tend to use round numbers (in one numbering system or another). There is no inherent problem in using non-round-ones.

Comment 11 Yuri Obshansky 2015-03-04 09:09:36 UTC
I think, the current solution did not fully resolve a requirement to set automatically JVM Memory Heap Size.
1. -Xmx and -Xms should be set to a multiple of 1024 
i.e. 1024M, 2048M, 3072, 4096, etc.
refer to http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1110613
2. What about -XX:PermSize=256m -XX:MaxPermSize=256m?
I suppose, that a permanent generation size should be increased accordingly to Heap Size. Definitely, it requires additional tuning to understand how many our application requires memory to load classes and static on start-up.
Need to take in account that summary memory used by engine will be calculate like -Xmx + MaxPermSize and permanent generation won't be garbage collected.

Comment 12 Yedidyah Bar David 2015-03-04 10:16:53 UTC
(In reply to Yuri Obshansky from comment #11)
> I think, the current solution did not fully resolve a requirement to set
> automatically JVM Memory Heap Size.
> 1. -Xmx and -Xms should be set to a multiple of 1024 
> i.e. 1024M, 2048M, 3072, 4096, etc.
> refer to
> http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1110613

This does not seem to talk about Java but about WebLogic. Also, it does not say which units are referred to, which seem to be bytes - [1] says:

-Xmxn
    Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB.

[1] http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html

> 2. What about -XX:PermSize=256m -XX:MaxPermSize=256m?
> I suppose, that a permanent generation size should be increased accordingly
> to Heap Size. Definitely, it requires additional tuning to understand how
> many our application requires memory to load classes and static on start-up.
> Need to take in account that summary memory used by engine will be calculate
> like -Xmx + MaxPermSize and permanent generation won't be garbage collected.

You are most welcome (really!) to do more benchmarking, report about your results, and provide a substantiated recommendation. If you do, please open new bugs as appropriate.

Comment 13 Sandro Bonazzola 2015-03-04 10:26:52 UTC
Moving back to ON_QA as per comment #12.

Comment 14 Yuri Obshansky 2015-03-04 11:01:04 UTC
(In reply to Yedidyah Bar David from comment #12)
> (In reply to Yuri Obshansky from comment #11)
> > I think, the current solution did not fully resolve a requirement to set
> > automatically JVM Memory Heap Size.
> > 1. -Xmx and -Xms should be set to a multiple of 1024 
> > i.e. 1024M, 2048M, 3072, 4096, etc.
> > refer to
> > http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1110613
> 
> This does not seem to talk about Java but about WebLogic. Also, it does not
> say which units are referred to, which seem to be bytes - [1] says:
> 

It doesn't matter.

> -Xmxn
>     Specifies the maximum size, in bytes, of the memory allocation pool.
> This value must a multiple of 1024 greater than 2 MB.
> 
> [1] http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html
> 

Right, the important sentence here is "multiple of 1024"


> > 2. What about -XX:PermSize=256m -XX:MaxPermSize=256m?
> > I suppose, that a permanent generation size should be increased accordingly
> > to Heap Size. Definitely, it requires additional tuning to understand how
> > many our application requires memory to load classes and static on start-up.
> > Need to take in account that summary memory used by engine will be calculate
> > like -Xmx + MaxPermSize and permanent generation won't be garbage collected.
> 
> You are most welcome (really!) to do more benchmarking, report about your
> results, and provide a substantiated recommendation. If you do, please open
> new bugs as appropriate.

Comment 15 Yuri Obshansky 2015-03-15 12:15:02 UTC
Successfully verified on RHEV-M 3.5.1-0.1 (build vt14).

Comment 16 Julie 2015-03-20 05:38:40 UTC
Hi Didi, 
   Just wondering what is the recommendation for'enough memory' for large setups?

Cheers,
Julie

Comment 17 Yedidyah Bar David 2015-03-22 07:25:58 UTC
(In reply to Julie from comment #16)
> Hi Didi, 
>    Just wondering what is the recommendation for'enough memory' for large
> setups?

The current recommendation iirc is 16GB for the engine machine, which (so I understand) should be enough for "large setups".

We currently do not give specific recommendations for specific setup sizes, just say that minimum is 4GB, recommended is 16GB. We might do some scale tests simulating various loads on various setup sizes, and then might give more specific recommendations.

Comment 18 Julie 2015-03-22 23:20:25 UTC
Thanks Didi.

Comment 19 errata-xmlrpc 2015-04-28 18:46:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0888.html


Note You need to log in before you can comment on or make changes to this bug.