Created attachment 977695 [details] logs Description of problem: I put cluster->memory optimization to 200% and define vm memory and guaranteed that above host free memory. Then when I start vm, start failed with libvirt error in vdsm log: Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 2264, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 3323, in _run self._connection.createXML(domxml, flags), File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2709, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: Unable to read from monitor: Connection reset by peer and with additional error in qemu log: Cannot set up guest memory 'pc.ram': Cannot allocate memory Version-Release number of selected component (if applicable): rhevm-3.5.0-0.28.el6ev.noarch How reproducible: Always Steps to Reproduce: 1. Set cluster memory optimization to 200% 2. Create vm that have memory and guaranteed memory above host free memory 3. Start vm Actual results: Start vm failed with libvirt error Expected results: Expect to some error message that guaranteed memory of vm above host free memory Additional info: We have memory filter, but it check that vm memory not above Max Scheduling Memory, that have value above of host free memory in case of memory optimization: <memory>8254390272</memory> <max_scheduling_memory>11976835072</max_scheduling_memory> So propose also to add checking if guaranteed memory not exceeds host free memory(minus some value, to leave some additional memory to host) engine log, have some nullPointerExceptions not really connected to this problem , have another bug for this
Memory optimization is based on ballooning and KSM, which reclaim memory from several VMs and allow running additional VMs. It is not possible to do this with a single VM, and the result will be swapping. So for a single VM we should not allow setting guaranteed (or other) memory for more than 100% of the host RAM. So the fix will be to limit this sort of a setting.
Following brainstorm meeting with rgolan this is the proposed solution. Distinguish VM memory assignment validation into 2 cases: 1. When the host is empty (no VM running). The upper memory limit should be host's available physical memory. 2. When the host is running one or more vms. The upper memory limit should include memory over commitment calculations. Here are sample cases for host having 4GB memory, With 150% memory over commitment. Case 1: Trying to run vm with 4.5 GB memory should fail. Case 2: Host is hosting a vm committing 3 GB memory. Trying to run another vm with 2 GB memory could succeed. Case 3: Host is hosting a vm committing 3 GB memory. Trying to run another vm with 3.1 GB memory should fail. If vm run fails due to insufficient memory. We should report the relevant host upper memory limit calculation (physical upper memory when hosting single vm, otherwise over committed upper memory limit). Another relevant consideration is reporting warning event when defining a vm (template) dedicated to host. With memory allocation beyond host's physical capacity. This will effect changes to all add VM commands and modify VM commands.
Scott, please review comment 2. Unless we're missing something, any VM with RAM size bigger than the physical host memory size will end up swapping. The only case which may avoid it is having a balloon taking out memory back to the host. However when the balloon will be deflated the host will start swapping. So all in all I recommend to prevent such cases where a VM has RAM size bigger than the host memory size. Thoughts?
Created attachment 986643 [details] insuficient memory fail message
Greetings team, Agree "Unless we're missing something, any VM with RAM size bigger than the physical host memory size will end up swapping." The question is when, how long and the ramification to the long run host operations? My general attitude is to trust the administrator to know what s/he is doing, even if s/he is less then wise/skilled. My role as a developer is to provide the means to fulfill vm administrators task. I should provide adequate warning and/then audit the misbehavior/damage. I should not prevent it (root is not prevented from rm the OS). In the same theme allowing administrator to assign more then 3GB to vm running 32bit OS is also resource waste. Therefore my inclination is: 1. Warning the administrator with interuptive message. Reporting wasteful/reckless memory allocation. (Different bug?). 2. Prevent allocating more then host physical mem for running single vm. As suggested in comment 2. 3. Improve mem allocation fail message with actual upper limit number (Different bug?). See attached pic comment 4. Your comments appreciated.
I'd stick with skipping overcommint in case that is the only VM. now a general question, we can set a more elaborated limitation on the admin by using the "free" mem on host to determine if that qemu proccess would be able to start. since no ksm and balloning is on it should be accurate enough I think. we can decide we do that for a single VM and continue the same as today otherwise.
After additional investigation, vm failed to run with above error message if I specify guaranteed_memory > free_memory + free_swap
(In reply to Artyom from comment #7) > After additional investigation, vm failed to run with above error message if > I specify guaranteed_memory > free_memory + free_swap we're not going to take swap into accont
3.5.1 is already full with bugs (over 80), and since none of these bugs were added as urgent for 3.5.1 release in the tracker bug, moving to 3.5.2
Created attachment 1016884 [details] vdsm and qemu log Checked on ovirt-engine-3.6.0-0.0.master.20150412172306.git55ba764.el6.noarch Have host with: total used free shared buff/cache available Mem: 3947152 252484 2464884 197772 1229784 3237136 Swap: 0 0 0 Try to run vm with memory=6000MB and guaranteed memory=1024MB under cluster Memory Optimization equal to 200% Start vm failed with the same error as above: Cannot set up guest memory 'pc.ram': Cannot allocate memory 2015-04-21 13:37:24.556+0000: shutting down So it seem we need to check if host vm have less memory(not guaranteed) than host free memory + swap
(In reply to Artyom from comment #11) > Created attachment 1016884 [details] > vdsm and qemu log > > Checked on ovirt-engine-3.6.0-0.0.master.20150412172306.git55ba764.el6.noarch > Have host with: > total used free shared buff/cache > available > Mem: 3947152 252484 2464884 197772 1229784 > 3237136 > Swap: 0 0 0 > > Try to run vm with memory=6000MB and guaranteed memory=1024MB under cluster > Memory Optimization equal to 200% > Start vm failed with the same error as above: > Cannot set up guest memory 'pc.ram': Cannot allocate memory > 2015-04-21 13:37:24.556+0000: shutting down > So it seem we need to check if host vm have less memory(not guaranteed) than > host free memory + swap The engine calculates available memory using minimal_guaranteed, and this is what you should be using. See: http://www.ovirt.org/Sla/FreeMemoryCalculation
When creating a new VM you have a "Resource Allocation" side tab. In this tab you have "Physical Memory Guaranteed" and this is the part that is being used for scheduling memory calculation as you can read in the link I provided in comment 12. If you set this one the a value greater than host physical memory, that VM will be blocked from starting.
Checked on ovirt-engine-3.6.0-0.0.master.20150519172219.git9a2e2b3.el6.noarch Host: total used free shared buff/cache available Mem: 16151584 526232 14711308 17184 914044 15286184 Swap: 0 0 0 Create new vm with guaranteed memory=16000MB Still receive error: Cannot set up guest memory 'pc.ram': Cannot allocate memory And no validation error from engine.
(In reply to Artyom from comment #14) > Checked on ovirt-engine-3.6.0-0.0.master.20150519172219.git9a2e2b3.el6.noarch > Host: > total used free shared buff/cache > available > Mem: 16151584 526232 14711308 17184 914044 > 15286184 > Swap: 0 0 0 > > Create new vm with guaranteed memory=16000MB > Still receive error: > Cannot set up guest memory 'pc.ram': Cannot allocate memory > > And no validation error from engine. Artyom please share the engine log in debug mode . I want to see how come we didn't fail the scheduling attempt
Created attachment 1086183 [details] engine log in debug mode
this is an automated message. oVirt 3.6.0 RC3 has been released and GA is targeted to next week, Nov 4th 2015. Please review this bug and if not a blocker, please postpone to a later release. All bugs not postponed on GA release will be automatically re-targeted to - 3.6.1 if severity >= high - 4.0 if severity < high
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
The underlying issue here is that QEMU has to be able to call malloc to get an allocation descriptor for the full VM memory. This call will fail if the free memory + buffers + swap amount is lower than the VM's memory. The fact that we try to start a VM on a host that does not have enough memory is pretty serious. We try again when that fails.. but we can select some other host with not enough memory. That is pretty much unacceptable especially in the case of HA VMs. So this is not about a nicer error message, but about the fact that we select a bad host during the scheduling. The fix is to split the memory check for the two cases - enough real memory for VM start vs. allowed overcommit.
This bug has target milestone 3.6.2 and is on modified without a target release. This may be perfectly correct, but please check if the patch fixing this bug is included in ovirt-engine-3.6.2. If it's included, please set target-release to 3.6.2 and move to ON_QA. Thanks.
Sorry for the noise, something went wrong while changing target release.
Verified on rhevm-backend-3.6.2.5-0.1.el6.noarch 1) host has 13Gb free memory + 2 Gb of free swap space = total 15Gb(30Gb of guaranteed) 2) Run vm with 16Gb(8Gb guaranteed) of the memory - Failed(validation error) 3) Run vm with 14Gb(7Gb guaranteed) of the memory - Succeed(host has total 13Gb of the free memory now and 23Gb free of the max scheduling memory) 4) Run vm with 12Gb(6Gb guaranteed) of the memory - Succeed(host has total 12Gb of the free memory now and 17Gb free of the max scheduling memory) 5) Run vm with 12Gb(6Gb guaranteed) of the memory - Succeed(host has total 11Gb of the free memory now and 11Gb free of the max scheduling memory) 6) Run vm with 10Gb(5Gb guaranteed) of the memory - Succeed(host has total 10Gb of the free memory now and 6Gb free of the max scheduling memory) 7) Run vm with 10Gb(5Gb guaranteed) of the memory - Succeed(host has total 9Gb of the free memory now and 1Gb free of the max scheduling memory) 8) Run vm with 8Gb(4Gb guaranteed) of the memory - Failed with validation message