Bug 1269774 - hotplug overcommit memory exceeding host memory cause qemu-kvm progress killed
hotplug overcommit memory exceeding host memory cause qemu-kvm progress killed
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 Linux
medium Severity high
: rc
: ---
Assigned To: Igor Mammedov
Virtualization Bugs
:
: 1432332 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-08 03:57 EDT by Yumei Huang
Modified: 2017-03-16 05:22 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-08-09 09:43:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yumei Huang 2015-10-08 03:57:52 EDT
Description of problem:
hotplug overcommit memory exceeding host memory,  cause the qemu-kvm progress killed.

Version-Release number of selected component (if applicable):
kernel: 3.10.0-321.el7.x86_64
qemu-kvm-rhev-2.3.0-28.el7

How reproducible:
always

Steps to Reproduce:
1. start a qemu-kvm progress,  /usr/libexec/qemu-kvm -name rhel7.2-test  -m 1G,slots=4,maxmem=4G -numa node   -vnc :1 -monitor stdio

2. hotplug 8G memory, which execeed the host memory.  
 (qemu)object_add memory-backend-ram,id=mem1,size=8G


Actual results:
qemu-kvm progress quit. the terminal  print "Killed".
(qemu) object_add memory-backend-ram,id=mem1,size=8G
Killed


Expected results:
qemu-kvm prevent the behavior, keep running, and print reasonable notice information.

Additional info:
doing regression testing
Comment 1 Yumei Huang 2015-10-08 04:18:33 EDT
correct qemu-kvm command line in comments 0(add mlock=on to cli):

/usr/libexec/qemu-kvm -name rhel7.2-test  -m 1G,slots=4,maxmem=4G -numa node   -vnc :1 -monitor stdio -realtime mlock=on
Comment 2 Yumei Huang 2015-10-08 04:50:06 EDT
I've test this with qemu-kvm-rhev-2.1.2-23.el7 and 2.3.0-2.el7, the bug appears in both version. So I think it's not a regression bug.
Comment 3 Igor Mammedov 2015-10-19 06:01:46 EDT
it works for me, object_add fails with "cannot set up guest memory 'mem1': Cannot allocate memory" error message

Pls, provide access to environment where it reproduces.
Comment 7 Igor Mammedov 2016-07-15 11:16:24 EDT
what is output of command:  sysctl vm.overcommit_memory
on the host where it happens?
Comment 8 Yumei Huang 2016-07-18 01:47:46 EDT
(In reply to Igor Mammedov from comment #7)
> what is output of command:  sysctl vm.overcommit_memory
> on the host where it happens?

The output is "vm.overcommit_memory = 0". 

Here are details:

qemu-kvm-rhev-2.6.0-12.el7
kernel-3.10.0-464.el7.x86_64

# free -m
              total        used        free      shared  buff/cache   available
Mem:           7822         116        7599           1         106        7555
Swap:          8063         119        7944

# /usr/libexec/qemu-kvm -name rhel7.2-test  -m 1G,slots=4,maxmem=4G -numa node   -vnc :1 -monitor stdio -realtime mlock=on
QEMU 2.6.0 monitor - type 'help' for more information
(qemu) object_add memory-backend-ram,id=mem1,size=10G
Killed

# sysctl vm.overcommit_memory
vm.overcommit_memory = 0
Comment 9 Igor Mammedov 2016-08-09 09:43:06 EDT
To avoid running into OOM in this case, you need to disable overcommit to get reliable error on allocation.
That probably would limit amount of host memory you could use (might vary depending on host) but it can't be helped.

Other options might be to use cgroups/ulimit to cap QEMU instance size but I haven't tried.

Anyway it's not something that qemu can fix/handle, it's the way kernel allocates memory by default and handles OOM conditions.
If you fill that it should be changed - reopen and reassign bug to kernel component.
Comment 10 Igor Mammedov 2017-03-16 05:22:33 EDT
*** Bug 1432332 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.