Bug 1188971

Hi Yedidyah,
I have 16 G RAM on engine server  
# cat /proc/meminfo #MemTotal: 16212064 kB
Up to bug fix description: Engine HEAP_MAX should be calculate such as
"1GB and 1/4 of available memory."
So, in my case = 1G + 4G = 5G
I see my engine process allocated with -Xms3958M 
Could you , please, clarify an algorithm?
BTW, Where it should be changed?
Cause I see old settings in file 
/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf
ENGINE_HEAP_MIN=1g
ENGINE_HEAP_MAX=1g

(In reply to Yuri Obshansky from comment #2)
> Hi Yedidyah,
> I have 16 G RAM on engine server  
> # cat /proc/meminfo #MemTotal: 16212064 kB
> Up to bug fix description: Engine HEAP_MAX should be calculate such as
> "1GB and 1/4 of available memory."

actually as "maximum of 1GB and 1/4 of available memory".

So for systems with <=4G, it will be 1G (as today), and for systems with >= 4G it will be 1/4 of available memory.

> So, in my case = 1G + 4G = 5G

16>4 => should be 1/4 of that, 4G

> I see my engine process allocated with -Xms3958M 

which is the result of the calculation, assuming the reported available memory was slightly less than 16G (which is normal).

> Could you , please, clarify an algorithm?
> BTW, Where it should be changed?
> Cause I see old settings in file 
> /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf
> ENGINE_HEAP_MIN=1g
> ENGINE_HEAP_MAX=1g

This it the "defaults" file.

It's written by engine-setup to /etc/ovirt-engine/engine.conf.d/10-setup-java.conf .

I think, should be 
"maximum of 1GB OR 1/4 of available memory".

Why -Xms3958M and not -Xms4096M?

(In reply to Yuri Obshansky from comment #4)
> I think, should be 
> "maximum of 1GB OR 1/4 of available memory".

Well, I am not a native English speaker. You are welcome to edit. Content Services will eventually rewrite this in better language.

(In reply to Yuri Obshansky from comment #5)
> Why -Xms3958M and not -Xms4096M?

See comment 3. The "available memory" is taken from the line "MemTotal" of /proc/meminfo .

(In reply to Yedidyah Bar David from comment #6)
> (In reply to Yuri Obshansky from comment #4)
> > I think, should be 
> > "maximum of 1GB OR 1/4 of available memory".
> 
> Well, I am not a native English speaker. You are welcome to edit. Content
> Services will eventually rewrite this in better language.
It is OK. That the reason why I started clarify your algorithm to be sure I understand it is correct. 
> 
> (In reply to Yuri Obshansky from comment #5)
> > Why -Xms3958M and not -Xms4096M?
> 
> See comment 3. The "available memory" is taken from the line "MemTotal" of
> /proc/meminfo .
Something wrong with calculation here:
MemTotal: 16212064 kB
16212064 / 4 = 4053016
So, should be -Xms4053M, isn't it.

(In reply to Yuri Obshansky from comment #7)
> (In reply to Yedidyah Bar David from comment #6)
> > (In reply to Yuri Obshansky from comment #4)
> > > I think, should be 
> > > "maximum of 1GB OR 1/4 of available memory".
> > 
> > Well, I am not a native English speaker. You are welcome to edit. Content
> > Services will eventually rewrite this in better language.
> It is OK. That the reason why I started clarify your algorithm to be sure I
> understand it is correct. 

Of course.

> > 
> > (In reply to Yuri Obshansky from comment #5)
> > > Why -Xms3958M and not -Xms4096M?
> > 
> > See comment 3. The "available memory" is taken from the line "MemTotal" of
> > /proc/meminfo .
> Something wrong with calculation here:
> MemTotal: 16212064 kB
> 16212064 / 4 = 4053016
> So, should be -Xms4053M, isn't it.

16212064/1024/4 = 3958.0234375

Perhaps we better write MiB/GiB and not GB [1], to be more accurate.

[1] http://en.wikipedia.org/wiki/Binary_prefix

(In reply to Yedidyah Bar David from comment #8)
> (In reply to Yuri Obshansky from comment #7)
> > (In reply to Yedidyah Bar David from comment #6)
> > > (In reply to Yuri Obshansky from comment #4)
> > > > I think, should be 
> > > > "maximum of 1GB OR 1/4 of available memory".
> > > 
> > > Well, I am not a native English speaker. You are welcome to edit. Content
> > > Services will eventually rewrite this in better language.
> > It is OK. That the reason why I started clarify your algorithm to be sure I
> > understand it is correct. 
> 
> Of course.
> 
> > > 
> > > (In reply to Yuri Obshansky from comment #5)
> > > > Why -Xms3958M and not -Xms4096M?
> > > 
> > > See comment 3. The "available memory" is taken from the line "MemTotal" of
> > > /proc/meminfo .
> > Something wrong with calculation here:
> > MemTotal: 16212064 kB
> > 16212064 / 4 = 4053016
> > So, should be -Xms4053M, isn't it.
> 
> 16212064/1024/4 = 3958.0234375
> 
> Perhaps we better write MiB/GiB and not GB [1], to be more accurate.
> 
> [1] http://en.wikipedia.org/wiki/Binary_prefix

Probably, you are right.
As far I know, everybody uses -Xms4096M option.
For example: https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/5/html/Performance_Tuning_Guide/sect-Performance_Tuning_Guide-Java_Virtual_Machine_Tuning-Large_Page_Memory.html

(In reply to Yuri Obshansky from comment #9)
> As far I know, everybody uses -Xms4096M option.

Of course, because people tend to use round numbers (in one numbering system or another). There is no inherent problem in using non-round-ones.

I think, the current solution did not fully resolve a requirement to set automatically JVM Memory Heap Size.
1. -Xmx and -Xms should be set to a multiple of 1024 
i.e. 1024M, 2048M, 3072, 4096, etc.
refer to http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1110613
2. What about -XX:PermSize=256m -XX:MaxPermSize=256m?
I suppose, that a permanent generation size should be increased accordingly to Heap Size. Definitely, it requires additional tuning to understand how many our application requires memory to load classes and static on start-up.
Need to take in account that summary memory used by engine will be calculate like -Xmx + MaxPermSize and permanent generation won't be garbage collected.

(In reply to Yuri Obshansky from comment #11)
> I think, the current solution did not fully resolve a requirement to set
> automatically JVM Memory Heap Size.
> 1. -Xmx and -Xms should be set to a multiple of 1024 
> i.e. 1024M, 2048M, 3072, 4096, etc.
> refer to
> http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1110613

This does not seem to talk about Java but about WebLogic. Also, it does not say which units are referred to, which seem to be bytes - [1] says:

-Xmxn
    Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB.

[1] http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html

> 2. What about -XX:PermSize=256m -XX:MaxPermSize=256m?
> I suppose, that a permanent generation size should be increased accordingly
> to Heap Size. Definitely, it requires additional tuning to understand how
> many our application requires memory to load classes and static on start-up.
> Need to take in account that summary memory used by engine will be calculate
> like -Xmx + MaxPermSize and permanent generation won't be garbage collected.

You are most welcome (really!) to do more benchmarking, report about your results, and provide a substantiated recommendation. If you do, please open new bugs as appropriate.

Moving back to ON_QA as per comment #12.

(In reply to Yedidyah Bar David from comment #12)
> (In reply to Yuri Obshansky from comment #11)
> > I think, the current solution did not fully resolve a requirement to set
> > automatically JVM Memory Heap Size.
> > 1. -Xmx and -Xms should be set to a multiple of 1024 
> > i.e. 1024M, 2048M, 3072, 4096, etc.
> > refer to
> > http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html#1110613
> 
> This does not seem to talk about Java but about WebLogic. Also, it does not
> say which units are referred to, which seem to be bytes - [1] says:
> 

It doesn't matter.

> -Xmxn
>     Specifies the maximum size, in bytes, of the memory allocation pool.
> This value must a multiple of 1024 greater than 2 MB.
> 
> [1] http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/java.html
> 

Right, the important sentence here is "multiple of 1024"


> > 2. What about -XX:PermSize=256m -XX:MaxPermSize=256m?
> > I suppose, that a permanent generation size should be increased accordingly
> > to Heap Size. Definitely, it requires additional tuning to understand how
> > many our application requires memory to load classes and static on start-up.
> > Need to take in account that summary memory used by engine will be calculate
> > like -Xmx + MaxPermSize and permanent generation won't be garbage collected.
> 
> You are most welcome (really!) to do more benchmarking, report about your
> results, and provide a substantiated recommendation. If you do, please open
> new bugs as appropriate.

Successfully verified on RHEV-M 3.5.1-0.1 (build vt14).

Hi Didi, 
   Just wondering what is the recommendation for'enough memory' for large setups?

Cheers,
Julie

(In reply to Julie from comment #16)
> Hi Didi, 
>    Just wondering what is the recommendation for'enough memory' for large
> setups?

The current recommendation iirc is 16GB for the engine machine, which (so I understand) should be enough for "large setups".

We currently do not give specific recommendations for specific setup sizes, just say that minimum is 4GB, recommended is 16GB. We might do some scale tests simulating various loads on various setup sizes, and then might give more specific recommendations.

Thanks Didi.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0888.html