Bug 1299523

Summary: Engine setup doesn't update the ENGINE_HEAP_MIN&ENGINE_HEAP_MAX default values if memory size changed..
Product: [oVirt] ovirt-engine Reporter: Nikolai Sednev <nsednev>
Component: Setup.EngineAssignee: Yedidyah Bar David <didi>
Status: CLOSED DUPLICATE QA Contact: Pavel Stehlik <pstehlik>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.6.2.5CC: bugs, didi, lveyde, nsednev, rmartins, sbonazzo, stirabos
Target Milestone: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-19 15:14:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikolai Sednev 2016-01-18 15:14:32 UTC
Description of problem:
Engine setup doesn't update the ENGINE_HEAP_MIN&ENGINE_HEAP_MAX default values if memory size changed.
I use engine deployed on a VM managed by another engine for upgrade testing.
Initially deployed my VM with 16 Gig RAM, so following the https://bugzilla.redhat.com/show_bug.cgi?id=1185411, ENGINE_HEAP_MAX=3889M:
cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf
ENGINE_HEAP_MIN="3889M"
ENGINE_HEAP_MAX="3889M"

Then I realized that we don't have much RAM on host on which engine-VM was running and decided to decrease the RAM to 6000MB for the engine-VM, I've made it while VM was powered-off. Then I powered on the VM and performed engine's upgrade at some point 3.5.6->3.6.7->3.6.1.3->3.6.2.3->3.6.2.5-0.1.el6.
During these upgrades I've suffered from slowness and saw 97% memory usage on VM, which was rising after each VM reboot or engine-upgrade.
The values within the cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf remained the same, although available memory for VM had been changed (decreased from 16Gig to 6Gig). I also have DWH and reports running on my VM, so their HEAP_MIN&HEAP_MAX memory allocations also remained the same as they were made during first clean deployment.

Each time that customer will change the total RAM for the engine, it should calculate total RAM available and change values for java engine, DWH and reports accordingly, either after running engine-setup or after each VM restart. Customer may use 3.6's feature memory-hot-plug and those values should also be changed accordingly, otherwise customer might get in to memory starvation like I did.  

Version-Release number of selected component (if applicable):
rhevm-3.6.2.5-0.1.el6.noarch
jasperreports-server-pro-6.0.1-1.el6ev.noarch
ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.5-1.el6ev.noarch
rhevm-dwh-3.6.2-1.el6ev.noarch
rhevm-dwh-setup-3.6.2-1.el6ev.noarch

How reproducible:
100%

Steps to Reproduce:
1.Deploy engine on VM with 16Gig RAM with DWH and reports.
2.Check for values returned from "cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf"
3.Decrease RAM for the VM.
4.Restart the VM.
5.Check cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf, it still has the same values as in step 2 (not changed).
6.Upgrade the engine and it's packages and then run engine-setup in order to upgrade the engine.
7.Check cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf, it still has the same values as in step 2 (not changed).
8.Increase RAM size for the VM and follow steps 4-7.

Actual results:
Once HEAP_MIN&HEAP_MAX values set during initial engine-setup, they never changed by engine-setup or VM restart on which engine is running, the same will be also for HW based hypervisor server (during server maintenance RAM increased/decreased and there is not automatic procedure for changing also HEAP_MIN&HEAP_MAX of engine&DWH&reports).

Expected results:
HEAP_MIN&HEAP_MAX of engine&DWH&reports should get changed automatically in case of RAM increased/decreased, unless customer manually setting those values, preferably via engine-setup itself, hence this requires from engine-setup to expose those values for manual changes, which are currently consealed.

Additional info:

Comment 2 Yedidyah Bar David 2016-01-19 15:05:28 UTC
(In reply to Nikolai Sednev from comment #0)
> Expected results:
> HEAP_MIN&HEAP_MAX of engine&DWH&reports should get changed automatically in
> case of RAM increased/decreased, unless customer manually setting those
> values, preferably via engine-setup itself, hence this requires from
> engine-setup to expose those values for manual changes, which are currently
> consealed.

Some ideas/thoughts:

1. Perhaps do that during engine restart instead of engine-setup.

2. Provide means to users to prevent us from automatically changing. Probably including retroactively (users that manually set it already), not sure how.

3. Add vars ENGINE_HEAP_PERCENT_MIN and ENGINE_HEAP_PERCENT_MAX . Use -Xms = max(ENGINE_HEAP_MIN, ENGINE_HEAP_PERCENT_MIN*physmem/100), similarly for -Xmx.

Comment 3 Yedidyah Bar David 2016-01-19 15:14:19 UTC

*** This bug has been marked as a duplicate of bug 1299526 ***