Bug 1025845
Summary: | MOM is ballooning all VMs at once, even VMs without free memory enough. | |||
---|---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Amador Pahim <asegundo> | |
Component: | mom | Assignee: | Martin Sivák <msivak> | |
Status: | CLOSED ERRATA | QA Contact: | Lukas Svaty <lsvaty> | |
Severity: | urgent | Docs Contact: | Cheryn Tan <chetan> | |
Priority: | medium | |||
Version: | 3.3.0 | CC: | abaron, acathrow, asegundo, bazulay, dfediuck, eedri, herrold, iheim, lpeer, mavital, msivak, rlandman, s.kieske, srevivo, vfeenstr, yeylon | |
Target Milestone: | --- | Keywords: | Triaged | |
Target Release: | 3.3.0 | |||
Hardware: | All | |||
OS: | Linux | |||
Whiteboard: | sla | |||
Fixed In Version: | mom-0.3.2-8.el6ev | Doc Type: | Bug Fix | |
Doc Text: |
When the hypervisor's memory pressure grows, MOM is supposed to reduce guests' memory to make more memory available to the hypervisor. But instead of selecting only the guests with free memory available, MOM reduced all guests' memory, so guests with high memory load had to use swap space. This issue is fixed with enhanced ballooning rules for computing the minimum available memory and reporting the swap usage of the guests. Now, MOM does not reduce guests' memory beyond their free memory limit.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1030515 (view as bug list) | Environment: | ||
Last Closed: | 2014-01-21 15:06:24 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | SLA | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1030515, 1031532, 1031534 | |||
Bug Blocks: | 1020228 |
Description
Amador Pahim
2013-11-01 17:52:49 UTC
Just a few things I'd like to clarify; 1. Once a VM starts swapping MOM should detect it and stop inflating it / start deflating. 2. The floor limit MOM is using is based on the "Physical Memory Guaranteed" settings we define for each VM in the Resource Allocation sub-tab of the new/edit VM dialog. Can you please provide the numbers set for these VMs? We are not getting swap information from the guest agent so this is definitely an issue. Because when we inflate the balloon, the VM will also put some of its data to swap and mom will then think that there is still enough reclaimable memory in the vm. To fix this we would have to modify the guest agent(s) - linux, win, mom collectors and vdsm policy for mom. There is also an error in the policy and we have a fix for that - http://gerrit.ovirt.org/#/c/19416/1/doc/balloon.rules (In reply to Doron Fediuck from comment #1) > Just a few things I'd like to clarify; > 1. Once a VM starts swapping MOM should detect it and stop inflating it / > start deflating. > > 2. The floor limit MOM is using is based on the "Physical Memory Guaranteed" > settings we define for each VM in the Resource Allocation sub-tab of the > new/edit > VM dialog. Can you please provide the numbers set for these VMs? Defined Memory: 2048 MB Physical Memory Guaranteed: 512 MB (In reply to Martin Sivák from comment #2) > We are not getting swap information from the guest agent so this is > definitely an issue. Because when we inflate the balloon, the VM will also > put some of its data to swap and mom will then think that there is still > enough reclaimable memory in the vm. My concern is also about all VMs being inflated/deflated with the same amount of memory, regardless their individual memory status. Is this an issue? Shouldn't MOM balloon VMs with different patterns considering its individual memory availability? > > To fix this we would have to modify the guest agent(s) - linux, win, mom > collectors and vdsm policy for mom. > > There is also an error in the policy and we have a fix for that - > http://gerrit.ovirt.org/#/c/19416/1/doc/balloon.rules > My concern is also about all VMs being inflated/deflated with the same
> amount of memory,
That is fixed in the referenced changeset.
The mom policy is designed to behave in one of two ways depending on the severity of host memory pressure. Under moderate pressure (between 5% and 20% free host memory) we try to balloon away only unused memory in guests. Under severe host pressure (< 5% free) we purposefully cause guest swapping in order to keep the host itself from entering a swap storm. Since you are observing guest swapping, could you share the state of host memory during that behavior? If the host has <5% free (counting Cached pages as free) then I would argue that the policy is behaving as designed. Adam: there are two issues there - Your fix http://gerrit.ovirt.org/#/c/19416/1/doc/balloon.rules is not present in the version he uses and that causes all the VMs to disregard the computed minimum in favour of the hard minimum (which is lower and the same for all VM). - When you have most of the memory in swap, MoM will think RAM is (almost) free and inflate the balloon because we do not have any info about the swap usage in the policy. But you are totally right about the two modes of ballooning we use. should this bug be on MODIFIED? if all patches are in, please move to ON_QA and mark fixed in is23 Unfortunately not, there are some patches that are still missing. The mom part is ready and vddm contains fixed policy. So it should behave better now. There are still situations where this won't be enough (mostly related to swap usage) and all the related bugs add support for dealing with it. mom stops to change size of balloon after first change change consulted with msivak, patch in process moving back to ASSIGNED MoM used the same variable stack for all policy runs. That caused old variable values to be used sometimes. mom-0.3.2-8.el6ev tested in Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0064.html |