Description of problem: Engine setup should update the HEAP_MIN&HEAP_MAX default values if memory size changed. I use engine deployed on a VM managed by another engine for upgrade testing. Initially deployed my VM with 16 Gig RAM, so following the https://bugzilla.redhat.com/show_bug.cgi?id=1185411, ENGINE_HEAP_MAX=3889M: cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf ENGINE_HEAP_MIN="3889M" ENGINE_HEAP_MAX="3889M" Then I realized that we don't have much RAM on host on which engine-VM was running and decided to decrease the RAM to 6000MB for the engine-VM, I've made it while VM was powered-off. Then I powered on the VM and performed engine's upgrade at some point 3.5.6->3.6.7->3.6.1.3->3.6.2.3->3.6.2.5-0.1.el6. During these upgrades I've suffered from slowness and saw 97% memory usage on VM, which was rising after each VM reboot or engine-upgrade. The values within the cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf remained the same, although available memory for VM had been changed (decreased from 16Gig to 6Gig). I also have DWH and reports running on my VM, so their HEAP_MIN&HEAP_MAX memory allocations also remained the same as they were made during first clean deployment. Each time that customer will change the total RAM for the engine, it should calculate total RAM available and change values for java engine, DWH and reports accordingly, either after running engine-setup or after each VM restart. Customer may use 3.6's feature memory-hot-plug and those values should also be changed accordingly, otherwise customer might get in to memory starvation like I did. Version-Release number of selected component (if applicable): rhevm-3.6.2.5-0.1.el6.noarch jasperreports-server-pro-6.0.1-1.el6ev.noarch ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch ovirt-engine-extension-aaa-jdbc-1.0.5-1.el6ev.noarch rhevm-dwh-3.6.2-1.el6ev.noarch rhevm-dwh-setup-3.6.2-1.el6ev.noarch How reproducible: 100% Steps to Reproduce: 1.Deploy engine on VM with 16Gig RAM with DWH and reports. 2.Check for values returned from "cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf" 3.Decrease RAM for the VM. 4.Restart the VM. 5.Check cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf, it still has the same values as in step 2 (not changed). 6.Upgrade the engine and it's packages and then run engine-setup in order to upgrade the engine. 7.Check cat /etc/ovirt-engine/engine.conf.d/10-setup-java.conf, it still has the same values as in step 2 (not changed). 8.Increase RAM size for the VM and follow steps 4-7. Actual results: Once HEAP_MIN&HEAP_MAX values set during initial engine-setup, they never changed by engine-setup or VM restart on which engine is running, the same will be also for HW based hypervisor server (during server maintenance RAM increased/decreased and there is not automatic procedure for changing also HEAP_MIN&HEAP_MAX of engine&DWH&reports). Expected results: HEAP_MIN&HEAP_MAX of engine&DWH&reports should get changed automatically in case of RAM increased/decreased, unless customer manually setting those values, preferably via engine-setup itself, hence this requires from engine-setup to expose those values for manual changes, which are currently consealed. Additional info:
Adding some related bugs to this RFE: https://bugzilla.redhat.com/show_bug.cgi?id=1185411 https://bugzilla.redhat.com/show_bug.cgi?id=1188972 https://bugzilla.redhat.com/show_bug.cgi?id=1113771
*** Bug 1299523 has been marked as a duplicate of this bug. ***
(copied from bug 1299523 comment 2) (In reply to Nikolai Sednev from comment #0) > Expected results: > HEAP_MIN&HEAP_MAX of engine&DWH&reports should get changed automatically in > case of RAM increased/decreased, unless customer manually setting those > values, preferably via engine-setup itself, hence this requires from > engine-setup to expose those values for manual changes, which are currently > consealed. Some ideas/thoughts: 1. Perhaps do that during engine restart instead of engine-setup. 2. Provide means to users to prevent us from automatically changing. Probably including retroactively (users that manually set it already), not sure how. 3. Add vars ENGINE_HEAP_PERCENT_MIN and ENGINE_HEAP_PERCENT_MAX . Use -Xms = max(ENGINE_HEAP_MIN, ENGINE_HEAP_PERCENT_MIN*physmem/100), similarly for -Xmx.
If a admin adds memory to the host I expect them to also change the default. I don't think we should have dynamic defaults, it can have unexpected results.
We're not exposing these settings in any way possible to the user, and we have them for DWH+engine+jasper-reports, we require documentation on this as well. Regarding adding/removing memory, what do you expect from customer to do if he'll remove the memory or if it'll be a VM with hot-plug RAM?
(In reply to Nikolai Sednev from comment #5) > We're not exposing these settings in any way possible to the user, and we > have them for DWH+engine+jasper-reports, we require documentation on this as > well. Open a bug to extend the existing one for anything missing. > Regarding adding/removing memory, what do you expect from customer to do if > he'll remove the memory or if it'll be a VM with hot-plug RAM? Do manual changes to JBoss to update memory amount.
(In reply to Yaniv Dary from comment #6) > (In reply to Nikolai Sednev from comment #5) > > We're not exposing these settings in any way possible to the user, and we > > have them for DWH+engine+jasper-reports, we require documentation on this as > > well. > > Open a bug to extend the existing one for anything missing. > > > Regarding adding/removing memory, what do you expect from customer to do if > > he'll remove the memory or if it'll be a VM with hot-plug RAM? > > Do manual changes to JBoss to update memory amount. Is there any option to reopen this bug? Customers will hit the same issue during their bare-bone->HE-VM migrations: I'm hitting this bug again i case of following the http://brq-setup.rhev.lab.eng.brq.redhat.com/ovirt-engine/docs/manual/en_US/html/Self-Hosted_Engine_Guide/chap-Migrating_from_Bare_Metal_to_a_RHEL-Based_Self-Hosted_Environment.html , during migration of engine from bare-bone host to HE-VM. I have initial ammount of 32Gig of RAM on bare-bone and during HE-VM deployment I'm chosing 4Gig for the HE-VM, because of the fact that hosted-engine-host has 32Gig and it have to share it with guest-VMs. Engine fails to get started on VM, as it takes the original MIN&MAX values of 8011M from /etc/ovirt-engine/engine.conf.d/10-setup-java.conf, received from the restored engine DB."Starting oVirt Engine: [FAILED]". Bottom line: After backup succeeded on HE-VM, it gained previous RAM values of 8011M from engine database, for MIN=8011M and MAX=8011M, so I had to manually edit the "vi /etc/ovirt-engine/engine.conf.d/10-setup-java.conf" and set both min and max to 1024M, otherwise engine-setup fails to start ovirt-engine "Starting oVirt Engine: [FAILED]".
(In reply to Nikolai Sednev from comment #7) > (In reply to Yaniv Dary from comment #6) > > (In reply to Nikolai Sednev from comment #5) > > > We're not exposing these settings in any way possible to the user, and we > > > have them for DWH+engine+jasper-reports, we require documentation on this as > > > well. > > > > Open a bug to extend the existing one for anything missing. > > > > > Regarding adding/removing memory, what do you expect from customer to do if > > > he'll remove the memory or if it'll be a VM with hot-plug RAM? > > > > Do manual changes to JBoss to update memory amount. > > Is there any option to reopen this bug? > Customers will hit the same issue during their bare-bone->HE-VM migrations: > > I'm hitting this bug again i case of following the > http://brq-setup.rhev.lab.eng.brq.redhat.com/ovirt-engine/docs/manual/en_US/ > html/Self-Hosted_Engine_Guide/chap-Migrating_from_Bare_Metal_to_a_RHEL- > Based_Self-Hosted_Environment.html , during migration of engine from > bare-bone host to HE-VM. I have initial ammount of 32Gig of RAM on bare-bone > and during HE-VM deployment I'm chosing 4Gig for the HE-VM, because of the > fact that hosted-engine-host has 32Gig and it have to share it with > guest-VMs. > Engine fails to get started on VM, as it takes the original MIN&MAX values > of 8011M from /etc/ovirt-engine/engine.conf.d/10-setup-java.conf, received > from the restored engine DB."Starting oVirt Engine: [FAILED]". Please add a docs bug to mind this in the guide. > > Bottom line: > After backup succeeded on HE-VM, it gained previous RAM values of 8011M from > engine database, for MIN=8011M and MAX=8011M, so I had to manually edit the > "vi /etc/ovirt-engine/engine.conf.d/10-setup-java.conf" and set both min and > max to 1024M, otherwise engine-setup fails to start ovirt-engine "Starting > oVirt Engine: [FAILED]".
Didi should we do something in restore for this issue in comment 7?
IMO no. The doc should say "The new engine VM should have the same amount of memory as the physical host we migrate from". We can also have a KB article saying how to decrease the default value.
Please open a docs bug on KB and guide.
Done- https://bugzilla.redhat.com/show_bug.cgi?id=1320579