Bug 1013758
| Summary: | Cgroups memory limit are causing the virt to be terminated unexpectedly | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Michal Privoznik <mprivozn> |
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.4 | CC: | acathrow, ajia, dallan, dyasny, dyuan, itxx00, jkortus, jmiao, lsu, mcsontos, mjenner, mzhan, rjones, rwu, thomlee, tlavigne, zhpeng |
| Target Milestone: | rc | Keywords: | Regression |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-0.10.2-28.el6 | Doc Type: | Bug Fix |
| Doc Text: |
Cause:
Previously, libvirt has contained a heuristic to determine the limit for maximal memory usage by a qemu process. If the limit was reached, kernel just killed the qemu process and hence the domain was killed as well. This, however, can't be guessed correctly. Never.
Consequence:
Domains were killed randomly.
Fix:
The heuristic was dropped.
Result:
Domains aren't killed by kernel anymore.
|
Story Points: | --- |
| Clone Of: | 891653 | Environment: | |
| Last Closed: | 2013-11-21 09:11:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 891653 | ||
| Bug Blocks: | |||
|
Description
Michal Privoznik
2013-09-30 17:41:06 UTC
I'm removing the heuristic that guesses the max mem limit for a qemu process. I've proposed the patches and hence I'm moving this to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-September/msg00967.html This bug can be verified, I rebuilt a qemu-kvm by my self and inserted a memory leak for check the memory limit which libvirtd set, just like:
int kvm_run(CPUState *env)
{
...
+ int num = 1024*1024*100;
+ void* ptr;
again:
+ ptr = malloc(num);
+ printf("malloc %p %d\n", ptr, num);
...
For previous libvirt-0.10.2-27.el6, the heuristic limit will cause qemu-kvm terminated earlier. The memory used before quiting is:
# free
total used free shared buffers cached
Mem: 8001120 2958312 5042808 0 31964 205540
-/+ buffers/cache: 2720808 5280312
Swap: 0 0 0
In libvirt-0.10.2-28.el6, it drops the heuristic limit, and OOM killed is not easy, qemu-kvm will use almost entire vm, then quit:
# free
total used free shared buffers cached
Mem: 8001120 7879944 121176 0 9948 64100
-/+ buffers/cache: 7805896 195224
Swap: 0 0 0
So I change the status to VERIFIED.
Since the patch set includes fix related to mlock,
- if (mlock)
- virCommandSetMaxMemLock(cmd, def->mem.hard_limit * 1024);
+ if (mlock) {
+ unsigned long long memKB;
+
+ /* VFIO requires all of the guest's memory to be
+ * locked resident, plus some amount for IO
+ * space. Alex Williamson suggested adding 1GiB for IO
+ * space just to be safe (some finer tuning might be
+ * nice, though).
+ */
+ memKB = def->mem.hard_limit ?
+ def->mem.hard_limit : def->mem.max_balloon + 1024 * 1024;
+ virCommandSetMaxMemLock(cmd, memKB * 1024);
+ }
so I did the following additional test:
1. start a guest with memoryBacking locked, and do not set hard_limit
# virsh dumpxml r6
<domain type='kvm' id='5'>
<name>r6</name>
<uuid>f14754c8-7d43-9808-6eba-e279c503713b</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<memoryBacking>
<locked/>
</memoryBacking>
...
we can see the 'Max locked memory' has been set according memory balloon size.
# cat /proc/`pidof qemu-kvm`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 62347 62347 processes
Max open files 1024 4096 files
Max locked memory 2147483648 2147483648 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 62347 62347 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
2. start a guest with memoryBacking locked, and set hard_limit to 2000000KiB
# virsh dumpxml r6
<domain type='kvm' id='5'>
<name>r6</name>
<uuid>f14754c8-7d43-9808-6eba-e279c503713b</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<memtune>
<hard_limit unit='KiB'>2000000</hard_limit>
</memtune>
<memoryBacking>
<locked/>
</memoryBacking>
...
The 'Max locked memory' has been set to hard_limit.
# cat /proc/`pidof qemu-kvm`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 62347 62347 processes
Max open files 1024 4096 files
Max locked memory 2048000000 2048000000 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 62347 62347 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1581.html |